Comparing Two Approaches for Estimating the Causal Effect of Behaviour-Change Communication Messages Promoting Insecticide-treated Bed Nets: An Analysis of the 2010 Zambia Malaria Indicator Survey

"These analyses serve to illustrate that BCC programmes can contribute to national programmes seeking to increase the use of ITNs inside the home. They also offer a viable approach for evaluating the effectiveness of other BCC programmes promoting behaviour that will reduce malaria transmission or mitigate the consequences of infection."

This paper compares two analytic approaches, propensity score matching and treatment effect modelling, in an effort to examine the relationship between exposure to behaviour change communication (BCC) messages and the use of an insecticide-treated bed net (ITN) the previous night.

The authors explain that the focus on combating malaria has been on distribution of ITNs to pregnant women as they partake in antenatal services and/or mass distribution of these bed nets to households. However, "there are several reasons for questioning the sole reliance on distribution approaches for increasing ITN use." Specifically: (i) though households may have an ITN, "a substantial number" reports not using them; (ii) "qualitative studies in many settings have consistently uncovered attitudes and beliefs unrelated to access and availability that contribute to decisions to not sleep under a bed net"; and (iii) anecdotal reports from the field reveal that households sometimes re-purpose usable ITNs for other household uses; hence, the potential role of BCC in promoting the effective use of ITNs.

However, there is a "challenge of evaluating BCC approaches using trials that randomly allocate individuals to treatment or control conditions....[I]ndividual preferences and opportunities often determine whether a person listens to a programme message on a mass media channel or participates in a group activity within the community. Since the characteristics that influence an individual's exposure to a programme message may also influence the behaviour targeted by that message, any observed difference in the behaviour between the treatment and control groups is potentially due to the confounding effects of these additional factors."

So, the authors use observational data here per these methods (outlined in detail within the paper):

1. Propensity score matching, which "uses exogenous background variables to create statistically identical intervention and control groups conditioned on those background variables. In this approach, exogenous background variables are regressed on a binary treatment exposure variable to calculate individuals' propensity to be exposed to the intervention." An example of an exogenous background variable: the individual-level characteristics of age of the respondent, measured in individual years.
2. Treatment effect models, which use "the predicted exposure to the intervention as an instrumental variable in a simultaneous equation that predicts the outcome behaviour of interest."

Data for this anaylsis were collected as part of the 2010 Zambia Malaria Indicator Survey (MIS). Among households owning at least one ITN, 3,380 women between the ages of 15 and 49 years of age were interviewed and were included in the analysis. "To measure exposure to BCC messages, all women were asked whether they had ever heard or seen any messages about malaria and, if so, how many months ago did they hear or see these messages....[T]he criteria for exposure was limited to women who reported hearing or seeing any malaria messages in the past six months and also cited at least one specific channel: television or radio, in the newspaper, on posters or billboards, or from peer educators and drama groups."

Findings, in brief, include: "When matched on similar propensity scores, a statistically significant 29.5 percentage point difference in ITN use is observed between exposed [to BCC messages] and unexposed respondents. Fifty-nine per cent of unexposed respondents reported sleeping under an ITN the previous night, compared to 88% of the exposed respondents. A smaller but similarly significant difference between exposed and unexposed groups, 12.7 percentage points, is observed in the treatment effect model, which also controls for the number of bed nets owned by the household and exposure to malaria information from health workers. Using either approach, a statistically significant effect of exposure to BCC messages on a woman's use of an ITN was found. Propensity score matching has the advantage of using statistically-matched pairs and relying on the assumption that given the measured covariates, outcome is independent of treatment assignment (conditional independence assumption), thereby allowing us to mimic a randomized control trial. Results from propensity score matching indicate that BCC messages account for a 29-percentage point increase in the use of ITNs among Zambian households that already own at least one ITN."

The discussion section reflects on these findings from an evaluation strategy point of view, noting that it is important to understand in advance the background factors that may influence exposure to a BCC programme's messages and include measures of these factors in the survey instrument.