Developing a Multidimensional Quality Index for enhanced process monitoring and supportive supervision of Behaviour Change Communication (BCC) interventions

Summary:

Why do some behaviour change communication (BCC) interventions fail to have an impact in some contexts but prove successful in others? Evidence is limited on the extent to which the answers lie in implementation issues versus external factors outside the implementers' control This article shares experience and learning from developing and rolling out a multidimensional quality index and accompanying set of benchmarks to measure and enhance quality of household- and community-level BCC interventions under the DFID- and EU- funded Suchana programme, aimed at improving the nutritional status of nearly a million people in North-East Bangladesh. The objective of the exercise was to develop a single, simple measure to capture multiple dimensions of quality of BCC activities. Initially, tools were developed for three key interventions: (i) individual counselling, (ii) group meetings, and (iii) growth monitoring & promotion. For each, the Index measures intervention quality across five-domains (capacity & supervision, preparation & environment, demonstrating appropriate skills, disseminating appropriate messages, and dedicating enough time) and calculates a score reflecting the proportion of observed sessions that met the standard across all five domains. Since launching the index in January 2019, the programme has seen approximately 60% increase in the proportion of activities meeting/exceeding benchmarks. Regular programme monitoring has also revealed upwards trends in outcome indicators such as exclusive breastfeeding and minimum dietary diversity for children, suggesting improved programme quality is contributing. However, further investigation is planned to assess the index's appropriateness and effectiveness in supporting these positive changes.

Background/Objectives

There is increasing evidence that BCC interventions delivered through trained community volunteers or frontline service providers may be unsuccessful even if service providers themselves have sufficient knowledge and skills. As such, the staff of the Suchana programme decided to move away from only tracking frontline staff capacity as a measure of intervention quality and attempt to measure the actual quality of sessions using a multidimensional index and context-specific benchmarks. The objective of this initiative was to develop a single, simple measure that captures multiple dimensions of quality, ultimately allowing the implementation with the ultimate goal of proving programme impact.

Description Of Intervention And/or Methods/Design

This article shares experience and learning from developing and rolling out a multi-dimensional quality index and accompanying benchmarks to measure and enhance the quality of household- and community-level BCC interventions. Initially, quality benchmarks and data collection tools were developed for three key interventions: (i) individual counselling; (ii) group meetings and (iii) growth monitoring & promotion. For each, the Index measures intervention quality across five domains (i.e. capacity & supervision, preparation & environment, demonstrating appropriate skills, disseminating appropriate messages, and dedicating enough time). To avoid data collection overload, data is collected during planned supervision visits through a modified supervision checklist. A digital data collection form eliminates the need for data entry and allows greater data quality assurance. During analysis. each domain was given equal weight in calculating the final index score, and the result was presented as the proportion of observed session that met the standard across all five domains.

Results/Lessons Learned

The data collection tools were jointly designed and field-tested with the technical and implementation teams, who were also involved in finalising the findings, to help build acceptance. In addition to providing an overall indicator of intervention quality, further breakdown of the index score allowed the implementation team to identify areas needing immediate attention. Essentially, this index was seen by the project team as a tool to 'break open the black box of implementation', providing a concrete basis for course-corrective decision-making. Since launching the index in January 2019, the programme has seen approximately 60% increase in the proportion of activities meeting/exceeding benchmarks. Regular programme monitoring has also revealed upwards trends in outcome indicators such as exclusive breastfeeding and minimum dietary diversity for children, suggesting improved programme quality is contributing.

Discussion/Implications For The Field

The MQI is a simple but rigorous measure to assess and enhance intervention quality. Additionally, linking findings with project-level monitoring data helps us understand how improved quality contributes to positive outcomes and impacts. The sector rarely undertakes quality monitoring with such breadth, hence we don't really know how many unsuccessful interventions have failed due to failures of implementation rather than design. The MQI supports focus on 'why' and 'how' something works, rather than simply 'what works', and enables implementers to demonstrate greater accountability, showing they are delivering what was promised.

Abstract submitted by: 

Md Masud Rana - Save the Children

Md Al-Amin - Save the Children

Sheikh Shahed Rahman - Save the Children

Mohammad Raisul  Haque - Save the Children

Alice Atkins - Save the Children