Consultancy – Final Evaluation for BHA Project in East Darfur, South Darfur and Khartoum 33 views0 applications

Background:

Approximately 14.3 million people in Sudan will need humanitarian assistance in 2022, 30% of the entire population. As of November 2021, there were over 3 million IDPs in Sudan, the majority of them (~1.75 million) in the Darfur region as a result of the conflict that started in nearly two decades ago in 2003. Despite substantial assistance in the Darfur region over the past decade, there continues to be a significant need as many areas remain prone to conflict and climate shocks. This proposed intervention will provide integrated, sustainable, and lifesaving WASH, health, and nutrition services to crisis-affected and vulnerable host community members and IDPs in East and South Darfur.

Project goal: The goal of the project is “To reduce the suffering and build the resilience of the most vulnerable IDP and host populations in Khartoum, East and South Darfur through integrated WASH, health, nutrition and multipurpose cash-based humanitarian assistance”.

Theory of Change (ToC) (statement): The project’s Theory of Change (ToC) draws on evidence from CARE’s long-term experience in Khartoum, East and South Darfur implementing humanitarian and development programs, as well as promising practices and lessons learned from recent interventions in the target areas.

Evaluation Purpose

The purpose of the end line evaluation is to learn from monitoring data and supplemental qualitative methods, as well as from end line data collection. During the end line evaluation, the purpose is to assess the extent to which the project was able to achieve its targets, the extent to which positive changes can be attributed to the project’s activities, how contextual factors affected implementation and the connection between actions and outcomes, the barriers and facilitators to success, and lessons learned and recommendations for how to improve the success of similar projects in the future. The evaluation will provide evidence and learning that can be applied in future projects, and to the accountability of the project to its affected population and stakeholders. Specifically, the evaluation sets out to:

1. Establish if the project achieved its set targets according to the approved Indicator tracking tables and explain any deviations from established targets.
2. To assess the relevance, efficiency, effectiveness, impact/changes, and sustainability of the interventions
3. To assess how the program ensured inclusion of vulnerable and marginalized communities and engaged with affected population and communities.
4. To document lessons learned/best practices (what worked well, what did not work well, what can be improved – practices that worked well during the project period?) and provide evidence-based recommendations for similar future interventions.

Scope of the evaluation:

Geographical coverage: The final evaluation will take place in the project implementation areas in eleven localities distributed in three states as below;

East Darfur: Assalaya, Abu Karinka, Bahar Al Arab, Ad Du’ayn, Sheria, and Yassin

South Darfur: Beliel, Gerida, and Kass localities and East and South Jebel Mara areas

Khartoum: Jabal Awlia and Umbada localities.

Evaluation Type

The end line evaluation for this project will be a performance evaluation that follows a pre-post design using mixed-methods data collection and analysis approaches.

The evaluator will apply the OECD/DAC criteria to assess the relevance, efficiency, effectiveness, and sustainability of the BHA project. The key evaluation questions include:

Relevance: Is the intervention doing the right things?

• Were interventions appropriate and effective for the target group based on their needs?
• Which target groups and individuals were reached by the interventions?
• How effective was the targeting approach in achieving the activity goal?

Effectiveness: Is the intervention achieving its objectives?

• To what extent do the activity’s interventions appear to have achieved their intended outputs and outcomes?
• To what extent do project intervention improved beneficiaries access WASH Health and Nutrition services.

Efficiency: How well are resources being used?

• How were problems and challenges managed?
• To what extent have the activity’s interventions adhered to planned implementation schedules?
• What was the level of efficiency and timely delivery of the goods or services?

Sustainability: Will the benefits last?

• To what extent did the activity take advantage of other USG and non-USG investments in the same target areas to facilitate linkages with complementary services, layering with earlier investments, and implementing an exit strategy?
• To what extent did the activity align and integrate with host government social protection strategy/policy/service delivery?
• Was the activity able to end operations at the close of the award without causing significant disruptions in the targeted communities?

Evaluation design and methodology

The evaluation is expected to employ a “mixed methods approach” that combines quantitative and qualitative techniques. The evaluation is expected to involve boys/girls, men/women, partners and stakeholders, field visits, and review of program document and program data. Data collection techniques may include desk reviews, key informant interviews, focus group discussions, satisfaction survey and observations. The team leader is expected to give due attention to the methods employed at baseline used to benchmark the performance of tracked outcome indicators. The survey will incorporate both qualitative and quantitative components, using, but not limited to, the following key data collection methods:

1. Desk review of project documents and other background documents like project proposal, log frame, assessment reports, etc
2. Survey to collect quantitative indicators that cannot be assessed through secondary data
3. Semi structured interviews with key informants and other community groups such as women and youth groups
4. Focus group discussionswith target women, girls, men, and boys, as well as community leaders.
5. Observations from the field – basic service provision, natural environment, community institutions, livelihoods activities, etc.

Any limitations to obtaining and verification of program data as well as to the methods and analysis should be clearly documented in the report. All efforts should be made to capture gender disaggregated data. The team leader is expected to refer to the OECD-DAC criteria, the Sphere Standards and BHA guidelines on Evaluating Humanitarian Actions.

Evaluation Approach

An independent external team leader has responsibility for the evaluation. Because of challenging travel conditions, the external team leader will be working remotely. The CARE MEAL team will provide support to the leader throughout data collecting. The team leader will assess the tools utilized for the baseline survey and make any necessary modifications.

For more accuracy and easy transferring data, quantitative data will be processed digitally using kobo toolbox. Team leader will be responsible from upload the household questionnaire to the Kobo system and provide training to the field team and enumerators. Additionally, he/she will do data analysis and generate the evaluation report. The CARE MEAL team will assist in collecting data in the field on behalf of the team leader due to the challenges of traveling to the specific states and localities.

The evaluation process will involve active participation from the project’s local partners, namely JMCO, SADO, NAHA, and SHOA. The evaluation team will gather information from the respective staff of these partners, who will serve as key informants. Additionally, a project staff member from each partner in the states will be involved in the data collection process in the field.

Secondary Data Analysis and Desk Review

The selected team leader will do secondary data analysis and a thorough examination of all project documents and reports in order to provide a foundation for reviewing and updating data gathering tools. Furthermore, project documentation and other reports will be utilized to assess the progress made on monitored output indicators and record the extent to which the interventions have met their predetermined targets. The review will offer potential justifications for any deviations from the established goals. CARE will provide the team leader with the essential project documentation and strategy documents to enable the evaluation. The desk review will also assess the degree of accomplishment of planned actions and the level of achievement of the established targets.

Mixed-methods Performance Evaluations: for this project CARE will conduct a performance evaluation which consist of both quantitative and qualitative data collection, which are systematically integrated. A final, mixed-methods performance evaluation must integrate a comparison of baseline and end line quantitative data, as well as a qualitative study. The qualitative study should be designed to explore issues identified in the quantitative results and answer evaluation questions that are beyond the scope of the quantitative survey (e.g., sustainability, management, etc.). Where possible, mixed-methods performance evaluation should pull from other sources of data including different project reports.

Population-based survey (PBS):

A population-based survey (PBS) will be conducted to assess changes in outcome indicators on access to safe drinking water, hygiene practices, access to safe and recommended latrines as well as access to optimum health services as these services are targeting the whole community. The PBS will follow a two staged sampling technique as detailed in the BHA M&E guidance document. The first stage of sampling will be the selection of villages from a sampling frame of all targeted villages in the catchment areas of targeted HFs that are also receiving WASH services from the program. The first stage will employ probability proportional-to- population size (PPS) sampling to select villages to be enumerated. The second stage of sampling will include the systematic random sampling of households from the selected villages. Focus Group Discussions and Key Informant Interviews. Sample size for the PBS will be based on the proportion of households targeted by the WASH program that are collecting all water for drinking, cooking, and hygiene from improved water sources.

Focused group discussions will be conducted with a number of groups to tease out and document community perceptions, areas of strength and areas of improvement in the different interventions under the project. Focused group discussions will be conducted with community members and community structures such as the water management committees and community development committees. Key Informant interviews will be conducted with line ministry management, local authority representatives, cluster-coordinators and community leaders to document the interventions’ strengths and gaps in technical designs, coordination and implementation especially considering timeliness, sequencing and soundness of the interventions.

Sample size determination and sample distribution:

This evaluation will use Glenn. I., 2002 method to determine the sample size, for more accuracy, sample size will be calculated separate for each state. The identified sample size will be distributed proportionally to different localities and locations.

Glenn. I., 2002 method to determine the sample size.

Sample size (n) = Total **HHs (N) / (1+N*r²) (**r is a margin of error (degree of accuracy).

The sector indicators will be measured with a margin of error of 5%. The total number of samples for each sector will be allocated respectfully among the states depending on the number of beneficiaries in these sectors in the three states. Below shows the sample size for each sector:

400 for WASH, 400 for health, 399 for nutrition and 384 for MOCA, including overlaps between different sectors.

Samples will be distributed 718 in SD, 551 in ED and 314 in Khartoum.

The whole sample will be allocated to the three states in proportion to the distribution of targeted beneficiaries in each sector. Therefore, the samples will be distributed to the states as follows:

Data Collection Tools and Procedure

The team leader and CARE will work closely together to update and develop the evaluation tools. The PBS surveys will to a larger extent adopt the baseline tools but with a few updates considering any new information required that was not needed or missed at baseline. FGDs and KIIs tools will be developed with the guidance of the team leader and considering evaluation questions/ matrix. All questionnaires and tools will be translated to Arabic and deployed in Arabic. CARE will hire and train enumerators and supervisors for collecting primary data. Quantitative data will be collected digitally through mobile phones using kobo toolbox.

Data Collection and Quality Assurance

The team leader will be responsible for ensuring data quality throughout the collection period. Prior to data collection, training will be provided to enumerators on the tools and field visit procedures. Survey, FGDs and KIIs tools will be pre-tested prior to the actual data collection to identify challenging questions and internalizing the questions. At field site, CARE MEAL team will conduct spot-checks and review completed questionnaires for completeness and accuracy. Every morning before data collection begins, enumerators will be given feedback on issues identified from the submitted or completed questionnaires. The final completed questionnaire will be signed off by the enumerators, supervisors, and MEAL officers.

Data Analysis and Report Writing

Quantitative primary data will be entered onto Kobo-collect online system. The external independent team leader will be responsible for data analysis and compiling the final report. Qualitative data will be analysed following content analysis methods or equivalent approaches.

In general. The methodology should be designed to mitigate against the numerous risks and challenges in the context, which will be discussed in more detail during inception phase, Finally, the consultant will be seeking to work to the principles of evaluation. Specifically, and not outlined/specified elsewhere in this scope of work.

• Independence: measures should be put in place to prevent bias.
• Usefulness: final findings must be articulated clearly and in a way that maximizes the potential for these findings to inform decision-making.
• Representativeness: final should strive to include a wide range of beneficiaries, including from different genders, age groups, ethnic groups, and locations (e.g., urban, and rural) as relevant to the project.
• Gender and protection sensitivity: final must be gender and protection sensitive and also, where possible, ensure to assess the intended or unintended effects of the project on gender roles and responsibilities and power relations. At the same time, the final must assess the protection risks that faced the target groups from men, women, boys, and girls.

CARE Tasks

In order to make the evaluation assignment successful and deliver expected activities within the deadline and high quality, the evaluation shall undertake the following key tasks:

• Facilitate meetings with key project staff of CARE, CARE management, and/or other stakeholders.
• Consolidate feedback on data collection tools from program quality team and finalize draft data collection tools to be tested.
• Facilitate training for the enumerators who will pre-test the data collection tools. If necessary, make final adjustments to data collection tools in consultation with the program quality team.
• Collect data from a representative sample of individuals from the target groups and key project relevant stakeholders using household questionnaires, key informant interviews (KII) and Focus Group Discussions (FGDs)
• Transcribe FGDs/ KIIs interviews.
• Organize and conduct training for enumerators focusing on data collection tools, methods, and overall field data collection process.

Individual Consultant/Firm Tasks

• Write the inception report including finalizing the evaluation methods and present it to the respective program quality and project team members.
• Review of baseline tools and update it as required.
• Upload of the quantitative tool in kobo collect system.
• Orientation of MEAL team in the field on the different tools and data collection strategy.
• Conduct data analysis Report writing and submission of first draft report.
• Presentation of findings and recommendations to and validation by key stakeholders.
• Finalize the report incorporating feedback and submission of final report.

There should be adequate female representation and participation throughout the data collection process. Where necessary, especially in rural areas, focus group discussions should be conducted separately for men and women and by data collectors of the respective genders. This arrangement will provide an opportunity for women to participate and share their insights and ideas freely.

Deliverables and Timeframe

The timing of the evaluation is expected to start in July 2024 – preparatory activities included and be completed data collection by 20th August 2024 with the delivery of the final report in 10th September 2024. Hence, the duration of the assignment is up to a maximum of 64 days. The important timeline for the key deliverables and milestones presented in the table below:

The evaluation report shall not exceed a maximum of 35 pages (excluding annexes). The evaluation report will be in English and submitted as an electronic copy (both PDF and MS Word format). The draft and final reports will have the following structure at a minimum.

• Title of evaluation: Abstract: Abstract of no more than 500 words briefly describing the
• Table of content:
• Acronyms
• Executive summary:
• Evaluation purpose and evaluation questions:
• Background:
• Evaluation methods and limitations: Findings:
• Conclusions:
• Recommendations:

Required Competencies from the Individual Consultant

• Individual consultancy.
• Advanced university degree (Masters / PhD) in International Development, Social Sciences, or any other related field with a minimum of 5 years of professional in international development and program evaluation.
• Demonstrated experience in assessments and/or evaluations of interventions on WASH, Health, Nutrition and Protection.
• Proven experience in data analysis, interpretation, and visualization
• Previous professional experience in Sudan/Africa is highly desirable.
• Excellent understanding of humanitarian and development issues.
• Advanced analytical and report writing skills.
• Proven and strong writing English language skills, with Arabic as a plus.
• Thorough understanding of different data collection methods.

Management of the consultancy and logistical support

The principal contacts for this consultancy will be the CARE Studies Evaluation Research and Learning coordinator. Under the guidance of the team leader, herein referred to as the consultant, CARE will conduct data collection and meet all costs related to data collection. The consultant will lead the exercise remotely and working closely with CARE MEAL team. During the implementation of this assignment, the evaluation team shall respect the terms and conditions of CARE policies and procedures on code of conduct, data protection and copyright, etc. The title rights, copyrights and all other rights of whatever nature in any materials used or generated under the provisions of this consultancy will exclusively be vested with CARE Sudan. All products developed under this consultancy belong to the project exclusively, guided by the rules of the grant contract. The consultant will need prior written permission to use any information from this evaluation for publication or dissemination.