Measuring and Understanding the Effects of a Performance Based Financing Scheme Applied to Nutrition Services in Burundi

Background

Malnutrition is a huge problem in Burundi. In order to improve the provision of services at hospital, health center and community levels, the Ministry of Health is piloting the introduction of malnutrition prevention and care indicators within its performance based financing (PBF) scheme. Paying for units of services and for qualitative indicators is expected to enhance provision and quality of these nutrition services, as PBF has done, in Burundi and elsewhere, for several other services.

The Nutrition PBF intervention

The intervention focuses on children under five years old. It follows the standard PBF model in Burundi and combines quantitative indicators to encourage an increase in service delivery (see below) and qualitative indicators. Quality of nutrition activities is assessed quarterly, and a bonus or penalty is applied to subsidies received by the facilities according to their quality score.

Table: Incentivized indicators

Community health worker (CHW) level

• nb of cases screened and referred to health center for acute malnutrition (AM)
• nb of classes promoting good nutrition

Health center (HC) level

• nb of cases screened and cared for severe and moderate AM
• nb of growth follow-ups

Hospital level

• nb of treated severe AM cases with complications
• length of the stay

All hospitals with nutrition services fall under the Nutrition PBF program. At lower levels, only HCs in the intervention group and the CHW that refer to them are subject to the Nutrition PBF.

Theory of change

The introduction of nutrition activities into the PBF program translates policy makers' belief that PBF can trigger some positive changes in the performance of the health personnel, facilities or system which will eventually impact on households and children. The investigators have identified seven tracks for transmission of effects for the health facility performance:

• The income track: the injection of extra financial resources might have a positive effect on nutrition services, as it allows the health facility manager to recruit more staff, to better equip his facility, etc.
• The cash track: the fact that the financial resources are transferred directly to the health facility's bank account allows the latter to rapidly and autonomously spend.
• The incentive track: the extra resources are conditioned upon higher performance in nutrition activities; this should motivate community actors and staff to improve their performance in order to boost the health facility's income and theirs (if bonuses are distributed among health workers). At facility level, the effect on other services is unclear: it can be negative for some (e.g. if the staff in charge of nutrition used to be responsible for other services which are now overlooked, as they are relatively less financially rewarding) and positive for others (if there are economies of scope - i.e. dedicating efforts to nutrition activities, reduce efforts required for other activities, thanks to synergies).
• The information track: through the contract, the fee system and the related information sessions, staff have a clearer view on what performance should be, as far as nutrition services are concerned. Feedback from the program may also guide their decisions to improve. The investigators hypothesize a positive effect on nutrition services. However, as for the incentive track, a negative effect could be that activities which are not remunerated may be perceived as non-important.
• Supervision \& enforcement track: under the new scheme, verification is extended to nutrition activities; this means that there will be more interaction between supervisors and the personnel in charge of nutrition. On top of the possible subsequent transfer of information (e.g. advice on good practices), the supervision may activate interpersonal motivators.
• Culture at provider level track: a PBF scheme invites health facility managers to develop a work culture more favorable to innovation, flexibility, responsibility and entrepreneurship. As PBF has been a national policy for five years, one can assume that this is already the case in Burundi. However, one cannot exclude that it could positively influence the nutrition department more.
• Health system track: it has been argued that PBF can trigger several system effects [1]. Here, part of these effects might come from the supervisors of the impact evaluation (e.g. the MoH requesting UNICEF and the WFP to better supply nutrition inputs; the WB solving some problems which may affect the study). Another part might come from the health facilities themselves (e.g. pressure upon the Department for Nutrition within the MoH to be a more reliable and responsive supplier). The investigators expect that community actors will refer more malnourished children to health centers and health centers will refer more severe acute malnourished children to referral hospitals. This may trigger some unexpected feedback loops.

Method

The research design consists in a mixed methods model adopting a sequential explanatory design. The quantitative component is a cluster-randomized controlled evaluation design: among the 90 health centers selected for the study, half receive payment related to their results in malnutrition activities (Nutrition PBF intervention group), while the other half get a budget allocation (Control group). Qualitative research will mainly be carried out at the end of the quantitative evaluation. The evaluation aims to provide the best estimate of the impact of the project on malnutrition outcomes in the community as well as outputs at the health center level (malnutrition care outputs) and to describe quantitatively and qualitatively the changes that took place (or did not take place) within health centers as a result of the program.

Data collection

Quantitative data collection consists in two rounds of health center and household surveys: baseline surveys before the implementation of the intervention, and endline surveys two years after.

The household surveys collect information on the nutritional and health status of each selected child, aged 6-23 months, as well as general information on their household (including socio-economics, food security indices).

The health facility surveys consist in various tools. To get information on malnutrition recovery rates, a total of 24 individual clinical files randomly selected among the files of all children under five years old enrolled in the moderate acute malnutrition (MAM) care program (12 files) and in the severe acute malnutrition (SAM) care program (12 files) during the last six months are transcribed. In addition, organizational aspects of the health centers as well as of the nutrition services are recorded through interviews to managers. To assess the quality of services, the investigators combine two techniques: patient-provider observation carried out on six pediatric consultations (performed by a maximum of two health workers) and exit interviews at the end of each of these observed consultations, in order to get information on the satisfaction level as well as to record anthropometrics of the children. Finally, to assess knowledge of the observed health workers, the investigators use vignettes to measure the practical knowledge on different tasks to perform: a pattern of a pediatric consultation is proposed and the health worker can ask all the questions (related to history and physical exams) necessary to arrive at a diagnosis and propose a treatment. Three vignettes are administered to every health worker observed in consultation.

During both survey rounds, lot quality assurance surveys are performed regularly by the field coordinator to assess accuracy of anthropometric data in the records. Most data are entered in "real-time", and irregularities detected and corrected by the field coordinator on a continuous basis. Data entry is done with the use of an electronic device. Android smartphones with Open Data Kit software and the ONA internet data management platform have been chosen for this purpose. The electronic data entry has the advantage of reducing risks of errors in recording the answers (thanks to automatic validity checks), and eliminating the need for double data entry from the paper to software transcription and to decrease considerably the time for transcription. Some questionnaires though need to be performed on paper (like the patient-provider consultation using an observation grid on paper): a double entry session is organized to avoid any entry error.

Sample size assessment

The sample size of the household surveys was computed on the smallest difference in the main outcome that can be considered of public health significance, i.e. a reduction of about 25% in acute malnutrition prevalence (2.5% points in absolute terms) in intervention centers' catchment areas as compared to control centers' ones. Assuming that the intervention will result in decreasing the prevalence of MAM in children aged 6-23 months from 10% to 7.5% [2], and assuming that 65 children aged 6-23 months will be surveyed in the catchment area of each health center, 90 health centers needed to be randomized to either the intervention or control group, for an α-error of 5% and a β-error of 20%. The number of children per health center was increased to 72 to allow for missing or incomplete data, amounting to a total of 6,480 children aged 6-23 months over the 90 selected health centers. In total, a sample of 6,480 children were surveyed for the baseline, and, two years after the start of the program, another sample of 6,480 children aged 6-23 months will be surveyed.

Selection of health centers invited to participate in the study has been done by simple randomization (computer-based random selection) among the 193 eligible health centers, i.e. health centers providing nutrition services (treatment of SAM and MAM). The 90 selected health centers have been paired on essential parameters of organization and functioning in relation to the outcomes (MAM rehabilitation activity, volume of activity, population in the catchment area, and percentage of recovery among malnourished children) as measured during the baseline survey. This will then be used to control for the potential confounding effect of these parameters. Within each of the 45 pairs, allocation to the intervention was done randomly with a lottery system organized during a workshop in December 2014.

The sample size of clinical files within the health center survey was computed on the smallest difference in the main outcome that can be considered of public health significance in intervention centers. Assuming that the intervention will result in increasing the recovery rate of acute malnutrition in children under-24 months from 80% to 90%, for an α-error of 5%, a power of 80% and an inter-cluster correlation (ICC) of 0.15 [3], a minimum of 12 clinical files per health center and per nutrition service (MAM and SAM rehabilitation services) were needed, among all children having been registered in the six previous months [4].

Analysis strategy plan

First, some descriptive analysis will be carried out in order to understand the main features of malnutrition management and of health services in general in Burundi at the health center level. Validation of the design will be performed with the baseline survey data by comparing the treatment group with the control group.

Second, with both survey rounds' data, the impact of the intervention will be assessed with multilevel statistical models with random effects at the health center level. Continuous dependent variables will be analyzed in mixed-effect regression models, whereas categorical ones (e.g. recovery yes/no) will be analyzed in logistic regression or Poisson regression models. At the population level, other factors of child malnutrition, such as household food security, socio-economic status, etc., will be controlled for. Interactions with season, child age and sex, stunting, and socio-economic parameters will be analyzed. An equity analysis will also be performed in order to understand whether the intervention benefits more the poorer or richer households. At the health facility level, other factors of child malnutrition recovery, such as for instance health facility staff' knowledge and know-how, will be controlled for. Interactions with child age and sex, stunting and MUAC will be analyzed.

Discussion

Although PBF schemes are blooming in low in-come countries, there is still a need for evidence, especially on the impact of revising the list of remunerated indicators. It is expected that this impact evaluation will be helpful for the national policy dialogue in Burundi, but it will also provide key evidence for countries with an existing PBF scheme and confronted with malnutrition problems on the appropriateness to extend the strategy to nutrition services.