End line evaluation/program review for Community health and social accountability program (CHASP) in Somalia 111 views0 applications

Background

Save the Children International (SCI) and Ministry of Health with funding support from the Swedish International Development Cooperation Agency (SIDA) and the Swiss Agency for Development and Cooperation (SDC) have been implementing Community Health and Social Accountability Programme (CHASP) project since May, 2017 to date. This project is being implemented in seven districts of Bari region, and the other three districts are Adado in Galgadud, and Afmadow and Kismayo in Lower Juba covering a total catchment population of 591,926 including 130,223 women of child bearing age and 118,385 children under the age of five.

The overall objective of the project is to ‘increase utilization of quality Reproductive, Maternal, New-born, Child Health (RMNCH) services which are accessible, acceptable, affordable and equitable through provision of Essential Package of Health Services (EPHS).

The objectives are expected to be achieved through the following outcomes:

Outcome 1: Evidence base of innovative solutions to inform replication and advocacy established

Outcome 2: Essential Package of Health Services, including nutrition, family planning and reproductive health, delivered to women and children.

Outcome 3: Health governance and management structures strengthened

Outcome 4: Community ownership, accountability and demand for health services enhanced

Purpose

The overall purpose of the evaluation is to assess the extent to which CHASP has achieved its outcomes/outputs and their contribution to reach defined goals of the project, and what factors have influenced/affected project delivery. In addition, the evaluation will examine how the decentralized and adaptive management approach, in a nexus context, has affected project delivery and how evidence creation has supported adaptions and otherwise impacted the project and the health sector. Lastly, the evaluation aims to document lessons learned and provide recommendations for future programming in fragile contexts/Somalia. The evaluation and its report will also achieve the purpose of being a learning document for Save the Children, Federal and member state government partners, implementing partners and for other stakeholders and donors.

Specific Objectives

• To assess the extent to which the intended outcomes have been achieved, and identify the internal/external factors that have had positive/negative influence on project delivery.
• To assess the extent to which the decentralized and adaptive management approach has affected project delivery, in a nexus context, as well as identify the supportive/hindering factors behind these, at both organizational and project level.
• To assess whether evidence generation activities were innovative and appropriate for the context, as well as the impact of the innovative evidence generated in informing project adaptions as well as wider policy and practice across Somalia.
• To determine the direction for future programming i.e. inform direction and areas to prioritize in the next phase and future programmes in similar contexts.

Evaluation criterion and questions

The evaluation questions will mainly focus on how well the CHASP interventions are being implemented, and how effectively contributing for the improvement of health systems and for the well-being of the population in every context concerned. The following questions shall be answered, but are not limited to:

1) Relevance and adequacy of the project design

• To what extent were project strategies and approaches relevant, valid appropriate to the different contexts and targeted populations?
• To what extent was the decentralized and adaptive management approach meaningfully applied, and which factors have/have not enabled this way of working?
• To what extent were evidence generation activities relevant, innovative and appropriate for the different contexts, and what is the existing or potential impact of the findings on future programming, policy and practice across Somalia? (Include evidence gathered in situational analysis).
• What are the major underlying factors (internal and external) and stakeholders influencing the achievement or non-achievement of the results?
• What benefits is this programme bringing to the local public health system in every concerned context, in the opinion of the district and regional authorities?

2) Effectiveness

• To what extent were the outcomes and outputs achieved according to plan in the four areas? Determine reasons for any shortfall.
• Assess how the capacity and ownership of project interventions by regional and district level MoH, and community-based structures have changed during the project period? What are the implications of these changes for future programming and what has SC’s role been in support this/any change?
• How has the governance structure of the federal-state/regional and district levels affected project implementation?
• Gender being a major cross cutting theme, to what extent was gender integrated in the project?

3) Efficiency

• What were the factors behind budget over- and underspends, and how justified were these?
• Whether the current governance set up for the project within SCI and SCS is efficient for the implementation of this type of project. Recommend areas of improvement (if any)

4) Connectedness and coordination

• How have the project activities been coordinated within different stakeholders to achieve overall objective?
• What and how have the project-identified stakeholders affected change in the project over the project period?
• What relationship(s)/interfaces existed between local health authorities and communities prior to the project and/or what relationships/interfaces were developed during the project period on health service demand, access and quality?

5) Sustainability

• To what extent are project results likely to be sustained, and why?
• Explore whether the activities implemented had smooth exit strategy or/and sustainability and make concrete recommendations for future projects to ensure the sustainability of systems and processes developed (including what has influenced the achievement or non-achievement of sustainability.)
• To what extent has the project contributed to building the health system in the areas of intervention?
• How has the capacity of relevant state actors to sustainably fund the health system in project target areas changed during the project period?

6) Impact

• What are the key short – and long term changes achieved by project, positive or negative and what are the key factors contributing to these changes?
• Is there any likely ability of the project or its components to be scaled up or replicated in other project areas and who would be the main actors/stakeholders to engage on scalability?
• Is there any “unexpected” outcome(s)/impact of the project?

Evaluation Methodology

The evaluation will adopt a participatory approach, and use mixed methods i.e. quantitative and qualitative research methods to collect and analyze the data. Evidence will be supported by providing full transcripts of informant’s responses in annex to the evaluation report. The focus will be to obtain an in-depth understanding of the relevance and effective system-building role of the CHASP project in the 10 districts, from the perspectives of the relevant stakeholders including MoH, selected beneficiaries’ groups, donors, SCI implementing staff and other sector-relevant actors in the concerned districts. Below are possible strategies (on limited to this) to be used for collecting the project’s level information;

• Desk review of project documents like, project funding document, field monitoring reports, project progress report, log frame, baseline report etc. MoH data, project data etc.
• Survey to collect quantitative indicators that cannot assess through secondary data.
• Interviews- Open ended questions regarding feelings, knowledge, opinions, experiences, perceptions on the proposed project strategies.
• Focus group discussions (FGDs)
• In-depth interviews with key or other community members
• Observations from fieldwork (heath facilities) descriptions of activities
• Case Studies
• Key informant interviews with MOH and SCI members/ stakeholders actors (such as local authorities, donors etc.) in the region.

Reviewers and Participants

This evaluation will be led by an external consultant with support from SCI. SCI support will involve logistics of data collection and site visits, providing background information, and managing any primary data collection that is deemed necessary by the consultant. The Technical Specialist and Research Manager at SCI Somalia will oversee the process of supporting the external reviewer(s) and draw on program managers and REALM managers in South central and Puntland to give maximum support. The project director, SCI senior managements and SC members will review and provide feedback to ensure quality of the report, and the findings are relevant to programming.

Timeline

The expected time-frame of this review is 40 working days, and the draft report will be submitted within two months of signing contract. SCI project director will collate feedback from stakeholders (SC members, donors and the government), and share with the consultant within two weeks of receiving the draft. The final report must be submitted after one week of receiving the feedback.

Quality assurance

SCI’s quality assurance process includes checks at various stages of the evaluation. As a first step the external reviewer(s) will take part of SCI’s Evaluation Report Quality Assessment Tool to guide the development of evaluations methodologies (also ensuring that ethical principles and child safeguarding guidelines and understood and followed throughout the evaluation process). The Inception report will be reviewed by SCI and SC Sweden, enabling feedback and dialogue with the external evaluation on the direction and progress of the evaluation and allowing for adjustments in approach if necessary and relevant. The quality of the final reports delivered under this TOR will be endorsed by team members from SCI & donors through a detail evaluation of the findings and recommendations as well as the methodological soundness behind those findings.

Ethical Consideration

The evaluation’s objectives will be clearly explained to all the respondents prior to gathering any form of information from them. Written consent of the respondents will be taken before collecting information where possible. The team will further be required to follow the SCI ‘Child Safe-guarding Policy’ and ‘Do No Harm’ policy through the evaluation process. The team will also need to comply with ethical considerations during information collection process with the girls and boys.

Work Plan

The detailed work plan for the evaluation will be finalized by September 30th, 2019 in consultation with the program team after a consultant is identified to undertake this evaluation. The roles and responsibilities to execute the process will be determined based on consultation with SC-Sweden/Switzerland.

Scope of work

The scope of work will be based on the objectives and key deliverables as specified in this ToR.

Key deliverables

• An inception report outlining the approach and methodology of the evaluation
• A draft report that addresses the expectations stipulated in the objectives and key questions of the evaluation
• A debriefing presentation for Save the Children International, where the overview of the findings and the recommendations will be discussed
• Final evidence based report as per objectives and evaluation questions stipulated in the ToR and following the reporting outline incorporating all feedback, suggestions and recommendations from Save the Children.
• Presentation of the final report to steering group meeting after sharing the report with the group members with 2 weeks to review

Report outline

The final report will contain the following elements:

1. Title Page
2. Table of Contents and Other Sections That Preface the Report
3. Executive Summary
4. Introduction and Background description
5. Rationale: (scope and purpose of research)
6. Methodology
7. Results Chapters or Key Findings.
8. Summary/Best Practices/Lessons Learnt, Conclusion and Recommendations
9. References and Appendices
10. List of references
11. Annex At the inception phase, we will share a detailed format with content to be considered under each chapter.

Expected Qualification and Experience

• At least master’s degree in public health or relevant disciplines
• 10 years of experience in similar evaluation of large health programs in context of or similar to Somalia
• Knowledge and demonstrated experience of designing and leading project review
• Demonstrated ability to think critically and strategically in giving practical advice from project reviews
• Fluency (verbal and written) in English and Somali is essential
• An evidence of previous work conducted (signed off report) specific for large institutional donors must be attached in the application
• The proposal may include a team instead of an individual. If a team is proposed, their roles should be clearly specified with the team leader stating his/her availability for any scheduled interview.