Evaluation of an MNCH Mentoring Intervention in Karnataka

By providing on-site mentoring for improved clinical care and service delivery, we hypothesize that the quality of services and care will improve and patients will have better clinical outcomes. Specifically we believe that routine on-site clinical and service delivery mentoring will strengthen clinical knowledge, skills and practices among staff nurses and ANMs (in year 2), will strengthen appropriate referrals and follow up care, will improve other aspects of service quality, and provider and client satisfaction. The evaluation will assess change during the first year of the intervention that focuses on clinical mentoring of staff nurses and on quality improvement in the PHCs as a whole. The evaluation will focus on success indicators as shown in Appendix 1. Due to the short time frame for evaluation, success indicators will be principally at the levels of inputs, outputs, and outcomes focusing on adherence to the Sukshema Critical Care package, although we also hope to assess some key health outcomes, in terms of reduction of "near-miss" cases.

1. Pre and post MM intervention evaluation strategy The "pre-intervention" evaluation will have two phases. The first phase will take place in all 122 sites before any intervention activities begin (Figure 1). This evaluation will focus on site readiness to provide quality services, provider knowledge, attitudes and reported or observed (or simulated) practices, supervision and mentoring, record keeping, and provider and client satisfaction. The NRHM SBA guidelines, PHC 24/7 guidelines, MHFW NSSK guidelines, and the Sukshema critical care package will all be used to determine expectations of clinical quality of care at this level.

The focus of the first year of the MM intervention is enhanced clinical skills in the intra-partum and postpartum periods and thus assessments of actual clinical practices are important. However, data on actual practices in the intrapartum and postpartum periods will be difficult to collect by observation during this baseline (especially those with complications). Most of the PHCs conduct only a few deliveries each week, and it will be therefore almost impossible to obtain a representative sample of observations. Therefore we will assess intrapartum and postpartum clinical practices largely though the audit of case sheets and registers. However, these case sheets are currently either not available or not used. Thus our strategy necessitates delay of this part of the second phase of the baseline until after the update training where all staff will be given and instructed in the use of new and improved case sheets. All sites will be given the opportunity to practice using these patient case sheets for one month before they are collected by project staff and used as a basis for assessing both compliance with the new case sheets, adequacy of reporting, and actual case management. Actual case observations will be done during routine MM visits where possible. The post intervention phase of the assessment will take place in all sites once year after the first (planned 6) MM visits and will repeat the baseline survey.

Pre and post MM intervention evaluation survey tools

The survey will use four types of survey tools pre and post MM intervention and one additional tool after one year in the intervention sites only:

A1. Questionnaires for all staff nurses on duty (including night duty) at the PHCs during the evaluation that will assess knowledge of intrapartum, postpartum, newborn care and infection prevention, and solicit provider perspectives on mentoring and supervision. In some cases, the questions will be in the form of case studies or asking for simulated demonstrations of practices.

A2. Facility audits and observations in all PHCs to assess site readiness to provide quality services and provide checks of drugs used: equipment and drugs, staffing and training levels, availability and correct use of case sheets, registers and referral slips. This tool will also have a section for observing practices in the postnatal ward.

A3. Postpartum client questionnaires with all women who have delivered up to one month before the survey traced to their homes that will assess client recall of information and services given and client satisfaction. Delivery registers will be checked and all women who delivered in the PHC n the previous month will be traced to their homes for interview.

A4. Case sheet and register audits for all cases for the one month before survey dates to assess completeness of reporting and appropriate client management A5. Focus Group Discussions with SNs in a sample of intervention PHCs: During the post MM intervention survey we will conduct focus group discussions (FGDs) with staff at a selection of intervention sites to understand their experiences with the MM intervention and to gather their suggestions for improvement.

Data collection: The field staff will consist of three supervisors per district, plus 13 teams of 2 interviewers, one of whom will manage the facility audits and staff surveys, and one the postpartum client questionnaires for each of the 122 PHCs. It will be necessary for each team to spend 2 days in each PHC, thus each team will be able to complete the survey work in 3-4 weeks. Simultaneous data collection will be carried out in both districts.

A week's training will be provided to the field teams that will include lectures on technical topics, demonstration interviews, mock interviews and field practice. The training will be provided to about 10% more persons than actually required for the survey, and the required number will be selected based on their performance during the training. KHPT and SJMC staff will conduct the training and will monitor the field work activities and quality of data.

Data processing: Double data entry will be done at KHPT, who will prepare a report of the pre-intervention survey and end-line survey comparison.

1. Routine continuous monitoring: First, project staff will routinely document staffing, training, financial and other project inputs, data that will be crucial for assessing the feasibility of scale up to other districts. During bi-monthly MM visits, staff will have a proscribed list of data to collect. As well as keeping details of staff at the sites their training background and any clinical mentoring given, the mentors will have a proscribed list of clinical skills to observe; wherever possible during their 3 day visits, they will observe women in labour, after delivery and in the postpartum ward, and where this is not possible, they will use models and case studies to assess staff skills. There will be continuous monitoring of the MM interventions through reports of MM visits that will include: documentation of visits through trip reports that detail facility meetings, talks given, happenings in the PHC or area and notes on any morbidity and mortality discussions; completed clinical competency checklists for mentored individuals. They will also photocopy the case sheets, which will have been used as a basis for discussion during the visit and facility action plans, for review by KHPT clinical staff. The case sheets will also be collected quarterly from the control sites by the KHPT district staff. These staff will encourage use of the case sheets at each of their quarterly visits without any additional training or mentoring.

Informed consent Verbal, witnessed informed consent will be sought from the head of the PHC who will be asked to give permission to survey the PHC; from staff nurses who will be interviewed in the PHCs and from postnatal women traced to the community.

Districts and sites Using a matched (for client volume) case control strategy, the MM intervention will be piloted and evaluated in PHCs in two districts, Gulbarga and Bellary that have a total of 122 24x7 PHCs (Gulbarga 70; Bellary 52). Sixty one PHCs (Gulbarga 35; Bellary 26) will be randomly assigned to an MM intervention or experimental group (stratified by taluka and 6-monthly patient delivery load), while the rest will serve as control sites.

SBA + training The first part of the intervention involves all the 122 PHC 24/7 facilities in the two districts for two reasons: first we feel that this training is generally needed and will help to put the intervention and control site staff on the same level playing field in terms of basic MNCH knowledge; second, we know from the Sukshema baseline survey conducted in 2011 that few staff use adequate reporting mechanisms such as case sheets. As these are needed both programmatically as job aids, and for programme evaluative purposes, we will use this training as an opportunity to introduce new case sheets. After the first phase of baseline survey (described below) and before the mentoring programme begins in the intervention sites, Sukshema project staff will conduct a 3 day training update (called SBA+) for all staff nurses and a one day SBA+ for all M.O.'s from all 122 PHCs in the two districts, irrespective of whether they have previously attended the NRHM Skilled Birth Attendant (SBA) training. The 3 day training will focus on the essential components on intra-partum and postpartum (mother and baby) care identified in the Sukshema critical care package that evidence shows can significantly reduce maternal and newborn morbidity and mortality. The one day training for doctors will focus on identification of danger signs, pre-referral management of complications, quality of care, case audits and improvement of management and documentation. At these trainings, staff will be given, and instructed in the use of enhanced case sheets for women and babies from labour through to discharge, including use of the partograph. After this training, there will be no further interventions in the control sites, except that Sukshema staff will visit every 3 months to collect copies of the case sheets.

Sampling strategy The PHC intervention and control sites will be selected systematically in each taluka, after stratification and matching by client volume per month.

MNCH mentoring In each district MNCH mentors (MMs), a new staff cadre, will be recruited and trained (Gulbarga 5; Bellary 4), each responsible for the intervention in 6-7 experimental/intervention PHCs. We will also train 3 additional MMs to act as a back-up (2 in Gulbarga and one in Bellary), total 12 mentors. The MMs will attend some of the PHC staff training but will also be trained for 4 weeks by project staff at St John's Medical College in essential clinical competencies and in how to mentor staff in clinical skills, team building and problem solving, service delivery improvement, as well as in how to be effective mentors. The MNCH mentors will be senior nurses, selected based on specific criteria such as clinical experience, training and mentoring abilities, and experience/familiarity with government health systems. They will be guided by a Mentoring Coordinator in each district who will support the mentors' activities and liaise between them, the PHCs, district government staff and KHPT district staff. During the first year, the mentors will be trained by the Project to (a) provide clinical mentoring in safe delivery to staff nurses by hands-on coaching; (b) provide mentoring in how to improve the quality of services by building teamwork and introducing problem solving initiatives to address other aspects of service delivery (including especially referral processes); and in year 2 to mentor ANMs who will come to the PHC during the MNCH mentor visits, in providing antenatal, postpartum and newborn care, and referrals and follow up.