Evaluating Behavior Change in International Development Operations: A New Framework

Affirming that putting people at the centre of the development process helps us understand their choices and constraints, institutions like the World Bank Group work with client countries on interventions that, at their core, aim at changing behaviour. To understand better the extent to which behaviour change is already built into World Bank Group practices and to help evaluators categorise observed behavioural barriers and interventions, the Bank's Independent Evaluation Group (IEG) developed this evaluative framework. The underpinnings of the framework are rooted in research - both standard neoclassical economics and behavioural economics - that shows that behaviour change is dependent on Communication, resources, Incentives and Information, Social factors and activities, and Psychological factors and activities (CrI²SP).

This paper serves three purposes:

First, it develops a definition of behaviour change that captures elements important to World Bank operations. In short, behaviour change activities focus on effecting a shift or change in shape of the demand curve for goods and services. Supply-side interventions, such as building latrines or building roads, can result in a change of behaviour if individuals use the latrines and roads. (Even when the supply of quality health services and nutrition resources exists, for example, several factors remain that may hinder the optimal use of these services, including lack of information, misinformation and harmful beliefs, inadequate resources to access the services, prejudice and fear, and individual trade-offs between health and pleasure. Evaluating World Bank's poverty-reduction projects with a behaviour change "lens" that focuses on the demand-side or at the beneficiary level allows for a deeper understanding of what works to improve service use when resources are not the binding constraint.

Second, the paper describes theories or frameworks of human behaviour that can be applied to behaviour change vis-à-vis international development interventions. As detailed here, the literature on behaviour (and, therefore, behaviour change) falls naturally into three categories: standard neoclassical economics; psychology and sociology; and behavioural economics, which is a combination of economics, psychology, and sociology. Building on these theories, several frameworks exist for helping policymakers design behaviourally conscious interventions. They are presented in the paper and include: Defra's 4E framework, the COM-B framework, the Messenger, Incentives, Norms, Defaults, Salience, Priming, Affect, Commitments, and Ego (MINDSPACE) and Easy, Accessible, Social, and Timely (EAST) frameworks, and the Focus, Opportunity, Ability, and Motivation Model in Sanitation (SaniFOAM) framework.

Finally, after establishing criteria for an evaluative behaviour change framework suitable for World Bank projects and finding that none of the existing frameworks fits that set of criteria, the paper proposes CrI²SP, which is designed to help evaluators assess systematically the degree to which projects:

• identify beneficiaries and their current behaviours;
• diagnose barriers to adopting a desired behaviour;
• design and implement a behaviour change intervention; and
• monitor and evaluate behaviour change to ensure midcourse corrections are made when needed and new project design improved.

Intended for sector evaluations to assess the prevalence and integration of behaviour change concepts into the life cycle of an intervention - from the diagnostic phase through monitoring and evaluation - CrI²SP includes 5 basic categories:

1. Communication - encompasses the channels or means through which behaviourally-informed messages are conveyed to the intended audience. Evaluating communication entails understanding whether the intervention effectively communicated the desired behaviours, and if so, how. A lack of information, misinformation, or incomplete or mistrusted information about an intervention can inhibit service take-up. A well-designed communication strategy can reduce informational frictions and encourage particular behaviours.
2. Resources may influence behaviour outcomes by expanding an individual's opportunity to engage in an activity or making the desired activity relatively less costly by increasing supply, thereby inducing movement along (down) the demand curve - but are not behavioural of themselves.
3. Were sufficient Incentives and Information put into place to support the desired behaviour change, and if so, how? Incentives and Information account for most shifts in the demand curve.
4. Social considerations influence people's behaviour through concerns about how they may be perceived by others. These often include social norms, which are broadly shared beliefs about what group members should and are likely to do. Social interventions might have engaged social norms through a soap opera that destigmatises public transportation or encouraged a group identity that has socially desirable attributes that could lead to better outcomes.
5. Psychological factors account for ways in which a project influenced individuals' behaviour through their nonrational or bounded-rational perception of the world around them. Mental models enter here (including stereotypes, causal narratives, or heuristics) as do cognitive biases and limitations (see appendix C).

The CrI²SP framework is subsumed as the central piece of a behaviour coding template that seeks to enable evaluators to capture, label, and categorise projects that will enable them to answer the following questions:

• Who was the intended population? What were their behaviours prior to the Bank's intervention?
• Why were potential beneficiaries acting in that way? What was done with the intent to change those behaviours?
• What was the result?
• How deeply were behaviour change considerations integrated into the project?
• Does the portfolio tend to integrate behaviour change activities into complex projects with multiple interventions, or to have single, focused behaviour interventions?

The behaviour change coding template itself is found in appendix B. Appendix C presents a user's guide that defines terms and gives examples for the behaviour change coding template; in particular, it details behaviour interventions by applying the CrI²SP framework. The CrI²SP framework and the coding template have been developed and refined as they have been applied to 33 World Bank projects. Examples of coded projects from water and sanitation, urban transport, and nutrition are given in appendixes D, E, and F. A set questions for use in semistructured interviews with project task team leaders (TTLs) is forthcoming.

The framework and template will be piloting in IEG's forthcoming water and sanitation and urban transit sector evaluations, as well as an evaluation on basic health services. In the process of applying these tools to those sectors' evaluations, the tools will continue to be shaped by the needs of the evaluation teams and so may evolve and change. CrI²SP is designed so that it can be applied to all sectors; however, sector-specific contours of the portfolio are expected. The sector evaluations will include an analysis of why Bank interventions may or may not be successful by looking through the lens of behaviour change.