Evaluating the Impact of Community Led Total Sanitation Programs in Mali

The evaluation study described involves 121 communities in the Region of Koulikoro in rural Mali. The intervention works in the following manner: communities are facilitated (by means of government and NGOs staff) to conduct their own appraisal and analysis of open defecation (OD) and take their own action to become open defecation free (ODF).

In Mali, the Open Defecation Free (ODF) status has been defined as follows: "each family has a latrine equipped with a cover that limits the proliferation of flies from the pits; all members of the family exclusively use such latrine to defecate; each latrine is equipped with a hand washing device (water + soap / water + ash bucket)".

To estimate the causal effect of CLTS the researchers need to construct a valid counterfactual in order to calculate what would have happened in the absence of the intervention. Random allocation ensures that on average, treated and untreated communities share the same observables and unobservables. Random assignment to treatment also overcomes the main selection problem found in evaluations, where those who are selected to receive the program may have different attributes than those who were not selected in the first place. These differences can be caused by observable attributes, more wealthy communities, more engaged leaders, better weather, etc, may be more willing to engage in CLTS programs, or by unobservable dimensions too. What is more important is that such differences can be affecting the outcomes the investigators want to measure. Random assignment to the program eliminates selection bias because it ensures that on average, communities receiving the program are similar to the ones that do not receive it.

Although random assignment is at the community level, the basic units of analysis of this evaluation are households. The investigators are interested in health outcomes for children under five, because diarrhea is among the main causes of child mortality. Also, the researchers are interested at looking at morbidity and school attendance for school age children. Finally, improved sanitation is supposed to produce a redistribution in the use of time at the household level. In addition the researchers are very interested at looking at variables that are directly related to the success/failure of the intervention. In particular, the investigators will monitor latrine use, water quality, general hygiene. The team will be able to determine whether lack of impact on health outcomes is due to lack of latrine use despite their availability, or whether it is due to lack of hand hygiene despite use of latrines.

UNICEF has observed in areas where the program has already been implemented that migration is relatively low, so the researchers do not expect much attrition. This decrease in diarrhea can be expected even if the village does not become fully ODF, but take up levels are lower.

The evaluation comprises gathering data at two points in time: a) baseline, before program implementation, b) follow up 12 months after program implementation in order to assess longer-term effects and sustainability. The investigators would be able to gather panel data at the community and at the household level.

While random assignment allows to compare average outcomes across communities, the investigators would also perform multivariate regression analysis in order to improve the precision of our estimates and control for any potential pre-treatment differences. Panel data allows the use of a difference in difference design, if necessary, and also to include initial (before the intervention) characteristics of households and communities. Standard errors will be clustered at the community level.

The communities included in the study understand and agree to be part of the study, meaning that they accept to work on sanitation issues with CLTS either right away or two years later. Randomization will be completed after baseline is conducted. UNICEF and the Directorate of Sanitation of Koulikoro (DNACPN) will conduct the triggering process in the 60 communities assigned to the treatment group.

One of the main concerns of random assignment is the potential contamination of the control group. This happens for example when there are interactions between members of CLTS communities and members of control communities. This is a problem in the presence of shared activities. The problem is that these interactions may cause changes in the control group. At the extreme, control communities and CLTS communities experience the same change, then the researchers will not be able to detect any effect. The researchers will ensure the study communities have geographic buffers, so that interaction is not expected to be very high. In order to check for interaction between family members living in different communities, the team added several questions in the surveys and document the extent of interactions. Another concern that often arises with randomized experiments is that control units may be receiving similar benefits from other interventions. The investigators will monitor control villages to ensure this does not happen and document this aspect of the design.

UNICEF plan to conduct strict monitoring during the intervention period (first 3 months). The research team plans to supplement this work by measuring relevant indicators of intervention compliance during the intervention period and after the end of the intervention.

The investigators will give careful attention to the variation in impacts across different groups, so treatment may be interacted with gender and age indicators, pre-existing characteristics of communities in terms of collective decision-making, among others in order to identify how these factors may explain why some people or some communities gain more than others from the program. Looking at heterogeneity in program impacts also helps in shedding light on the mechanism behind program's success (or failure).

This is one of the first evaluations using impact evaluation techniques with quantitative data [and random assignment] of CLTS programs in the developing world. It will also complement already existing evidence. Another advantage of this evaluation is that it will look carefully at behavioral outcomes that are behind the adoption of better sanitation practices and that are often overlooked in evaluations related to sanitation, which tend to focus more on health outcomes. It is widely accepted that better sanitation improves health, yet there is still much debate over what a cost-effective way to deliver a sanitation intervention may be. Success in delivery will very much depend on whether the program is able to identify bottlenecks that impede adoption of better sanitation practices and whether it is able to solve the issues that are identified.