CONSULTANCY SERVICES FOR END LINE EVALUATION; Community Based Early Warning & Early Response Project In Jubland & South West States Of Somalia 66 views0 applications

ENDLINE EVALUATION TERMS OF REFERENCE: COMMUNITY BASED EARLY WARNING AND EARLY RESPONSE PROJECT IN JUBLAND AND SOUTH WEST STATES OF SOMALIA

ABBREVIATIONS

DME: Design Monitoring and Evaluation

EWC: Early Warning Committee

EWER: Early Warning & Early Response

FEWS NET Famine Early Warning Systems Network

FGD Focus Group Discussion

FSNAU Food Security and Nutrition Analysis Unit-Somalia

GIS Geographical Information Systems

IDP Internally Displace Person

KII Key Informant Interviews

OECD Organization for Economic Co-operation and Development

RNG Random Number Generator

SomReP Somalia Resilience Program

ToR Terms of Reference

WVS World Vision Somalia

WFP World Food Program

WVG World Vision Germany

1.0 INTRODUCTION

1.1 Evaluation Summary

Project name: Community Based Early Warning and Early Response Project in Jubaland and South West States of Somalia

Project objective: Hunger crisis affected communities have increased capacity to predict, prepare and respond to natural disaster to contribute to effective recovery and increased resilience.

Project outcomes

Outcome 1: Improved community capacity to identify, report and respond to early warning identifiers

Outcome 2: Strengthened government and partner capacity to analyse, manage and utilize early warning information

Outcome 3: Community Disaster Management Plan Available and Operational.

Target beneficiaries: Direct Beneficiaries: 564 (470 EWC members, 47 village elders, 24 line ministry staff, 16 district staff, 7 humanitarian agency staff (SOMREP & WV Partnership project partners)

Indirect Beneficiaries: 180,000 (91,260 MALES, 88,740 Females, 46% below 15 years old, 14,7 IDPs)

Total = 180,564

Project locations: Bay region (Baidoa), Bakool (Wajid); Gedo (Luuq & Dollow)

Project duration: 15 months (October 1^st 2018 – December 30^th 2019)

Evaluation type: Endline Evaluation

Evaluation purpose

The purpose of the endline evaluation is to document and inform the key project stakeholders (donors, partners and beneficiaries) of the project progress with reference to the OECD/DAC evaluation criteria on relevance, effectiveness, efficiency, sustainability and impacts in relation to project results. The endline evaluation will also help to draw key lessons learnt and the best practices to the project stakeholders.

Methodology

The endline evaluation will adopt mixed methods design including quantitative and qualitative techniques as summarised below:

Quantitative

o Beneficiary household surveys

Qualitative

o Focus group discussion (FGD)

o Key Informant Interviews (KII)

o Document Reviews and Case studies

Evaluation time: 15^th January-30^th January 2020

2. Description of Projects Being Evaluated

Currently, several actors, such as FEWS NET, FSNAU, SomReP and WFP have set up early warning systems and collect information in Somalia. However, each stakeholder collects different data sets, with different areas of emphasis, over different timespans. FEWS NET, FSNAU and WFP collect trend data on specific indicators over large geographic expanses while WV-led SomReP narrows down, combining secondary information from FEWS NET and FSNAU, to produce a more specific “snap shot” of the humanitarian context within a specific geography. The fragmented nature of the existing early warning system leads to disconnects between early warning data, messaging to donors and consistency with regards to early actions triggered by this information. A coordinated Early Warning/Early Action system for Somalia would assist in timely mobilization of resources and consistency of approach between actors.

The proposed interventions will look at three aspects in early warning early action: 1) information gathering, 2) data analysis and 3) advocacy and action. Information gathering will mainly take place at the community level. WV Somalia will train and support Early Warning Committees (EWCs) to monitor community formulated early warning (EW) indicators and report their observation through mobile phone that will link to a central data platform. The proposed system will be configured on ArcGIS mobile leveraged through Survey123 application Survey123 for data capture. The gathered data will automatically populate itself in the ArcGIS online server for analysis and visualization using and this will be deployed on Operations Dashboard for reporting and decision making. Each EWC will be equipped with one mobile phone, entrusted to a literate champion identified by the committee who will serve as the focal point of the whole process; the project will also procure and provide 3G internet modems to each group to facilitate data transmission. WV Somalia will work with the communities to ensure the composition of the committee will be representative to all groups of the communities, including women, elderly and minority. The committee will endeavor to achieve equal representation of men and women (50% to 50%). In addition the project proposes to implement remote water level monitoring in shallow wells and boreholes using water level logger technology. The water level loggers will be installed at water sources such as boreholes and shallow wells to capture water levels and quality. This information is crucial in enabling timely intervention to address two constant threats to Somalis: water shortage and flash floods (especially along the rivers). The proposed project also intends to set up mini weather station at each of the targeted districts in order to have more accurate rainfall and temperature data that can be used to link to changes in ground water levels and other seasonality issues. The project will use the weather data and water level observations to calibrate existing hydro-geological models to better simulate water levels within. Data gathered from facility level such as school attendance data, disease surveillance data and nutrition screening data will be linked to the system to provide real-time and comprehensive information to inform disaster triggers. In the system, the project intends to build in thresholds that automatically facilitate decision making based on community Disaster Management Plans

2.1 Project Objective

Hunger crisis affected communities have increased capacity to predict, prepare and respond to natural disaster to contribute to effective recovery and increased resilience.

Project Indicators:

Impact:

• Of community actions taken as a result of data summaries from early warning system
• Proportion of target communities reporting improvement in disaster preparedness

Outcome 1: Improved community capacity to identify, report and respond to early warning identifiers

• Increase in the number functional community-based early warning systems in place (data disaggregated by communities targeted) during and at the end of the project
• of community-level early warning committees strengthened

Outcome 2: Strengthened government and partner capacity to analyse, manage and utilize early warning information

• # of issues brought to the agenda of stakeholders for decision making
• A functional Early warning system with tools, triggers and responses
• Outcome 3: Community Disaster Management Plan Available and Operational
• Proportion of communities (especially farming and pastoral) know the early warning signs and know what to do in case of an emergency or disaster

Output 1.1 indicators

• Increased knowledge of Early Warning and early action system for project staff, government ministries, humanitarian partners and community early warning committees
• # of early warning committees and District Authorities trained to be able to mitigate vulnerability to shocks and strengthen community early warning mechanisms and response

Output 1.2 Indicators

• Functional remote hydrological (water level and quality) and meteorological monitoring (Data collection, processing and analysis) stations
• · # of remote hydrological and meteorological monitoring stations established and functional

Output 2.1 Indicators

• Government and partner staffs trained on early warning information management
• · # of government staff and partners trained on early warning functionality
• · # of government staff with ability to conduct advanced data processing, analysis and reporting

Output 3.1 Indicator

• Early warning indicators, thresholds and response plans developed
• · # of Communities supported to develop own contingency resources tied to early warning indicators
• · # of communities with clear early response plans linked to their early warning indicators

3. Evaluation Target Audiences

The endline project evaluation is intended to benefit multiple stakeholders that have been involved directly or indirectly in the project implementation process. In particular, the following are the key stakeholders that will be involved in the evaluation process;

• Project beneficiaries including IDP and host communities in Baidoa, Wajid, Luuq and Dollow Districts
• SOMREP & WVS project partners
• Early Warning and Early Action Committees
• Ministry of Planning, Ministry of Education, Humanitarian Affairs and Other National and International Agencies operating in the area
• World Vision Germany

4. Evaluation type

This is an end of project evaluation that is aimed at assessing the progress made by the project towards achieving the project objective of hunger crisis affected communities increased capacity to predict, prepare and respond to natural disaster to contribute to effective recovery and increased resilience.

5. Evaluation Purpose and Objectives

The primary purpose of this evaluation is to assess the impact, appropriateness, effectiveness, efficiency and sustainability of the EWER project. The project endline evaluation will also help to draw key lessons learnt and the best practices to the project stakeholders. In particular, the project evaluation will be shaped by the following specific key evaluation questions:

Objective and Key Evaluation Questions

Impact: What has been the impact of the project interventions on the community?

In addition, establish the level of impact on the target beneficiaries.

Relevance

• Community involvement and participation in the design process, goal setting, planning and implementation.
• How equitably has the project benefited the; women, men, boys and girls, returnees, IDPs and host communities in general?
• The relevance and appropriateness of project design to the needs of the community.

Effectiveness

• What are the achievements against set objectives?
• Compare actual with planned outputs and how have outputs been translated into outcomes.
• The evaluation shall also establish the possible deviation from planned outputs and likely outcomes.

Efficiency

• How adequate were the available resources qualitatively and quantitatively?
• Were all the project resources utilised optimally?
• Explore alternative low-cost approaches that could have been used to achieve similar results?
• How could the efficiency of the project be improved without compromising outputs?
• Assess the timeliness of implementing the project activities.
• How adequate were the reporting and monitoring systems of the project?
• Have the project outputs been achieved at a reasonable cost?

Sustainability

• Are there sustainability plans, structures and skills in place to ensure there is sustainability of project benefits? How adequate are they?
• How is the community and local partners prepared to continue with the project outcomes?
• How likely are the outcomes to be sustainable and enduring? In what ways will it leave a legacy for its beneficiaries and the communities?
• In what ways are women and men in communities, the local partners and government stakeholder’s partners prepared to continue with the project outcome?

6. Evaluation Methodology

The evaluation methodology will be designed in alignment with World Vision’s Learning through Evaluation with Accountability and Planning (LEAP) guidelines and principles. To ensure the quality of evidence, the evaluation will be designed with reference to the Bond Evidence Principles Checklist. Specific reference will be made to the 5 key dimensions of voice and inclusion, appropriateness, triangulation, contribution and transparency.

The data collection process will apply both quantitative and qualitative methods. A detailed evaluation methodology will be designed by the external Consultant in consultation with WV Design Monitoring and Evaluation Manager, WASH & Infrastructures Manager and WV Germany who validate the sampling strategy and procedures.

The detailed design of methodology must include the following;

• The evaluation design
• Sampling for qualitative and quantitative surveys
• Data collection instruments, protocols and procedures
• Procedures for analysing quantitative and qualitative data
• Data presentation/dissemination methods.
• Report writing and sharing etc.

The key data collection methods will include the following among others.

• Document reviews including the project proposal, baseline reports, quarterly and semi-annual reports, midterm evaluation report, monitoring reports and project review reports.
• Focus Group Discussions (FGD) involving primary project participants and
• Key Informant Interviews with the line ministries, district authorities, Reflection and feedback sessions with staff and partners.

The Consultant will be expected to employ mobile data collection using smartphones leveraged on Kobo toolbox. Also, Geographical Information System (GIS) solutions will also be employed in the evaluation process; ranging from data collection, analysis and presentation of results.

7. Evaluation Deliverables

The Consultant will be expected to deliver the following outputs:

• An inception report detailing the approach and methodology to be used and sample size calculations, a detailed execution plan, data-collection tools.
• Draft report submitted to WV Somalia within an agreed timeline between the WV Somalia and the Consultant
• Presentation of the key findings and recommendations to WVS in Jubaland and South West States.
• All indicators must be presented overall and disaggregated by sex and disability status, where appropriate.
• Collected data (raw) after analysis complete with variable labels and codes, and the final evaluation tools submitted to WV Somalia and alongside the final report.
• Final report (soft copy) and 3 hard copies submitted to WV Somalia Quality Assurance team and WASH & Infrastructures Manager.
• The Consultant should note that the Final Evaluation Report shall follow the structure below customized from the UNDP (2009) Handbook on Planning, Monitoring, and Evaluation for Development Results.
• The evaluation report will also be guided by Bond Evidence Principles Checklist. Specific reference will be made to the 5 key dimensions of voice and inclusion, appropriateness, triangulation, contribution and transparency.
• The Consultant will be required to prepare and submit an executive brief of the evaluation report with infographics summarizing the key project achievements, recommendations, lessons learnt and the best practices.

7.1 Evaluation Report Structure

Title and Opening pages (front matter)-should provide the following basic information:

i. Name of the project evaluated

ii. Time frame of the evaluation and date of the report

iii. Project location (districts and country)

iv. WVS and donor logos as well as partner organisations

v. Acknowledgments

Table of Contents-including boxes, figures, tables, and annexes with page references.

List of acronyms and abbreviations

Executive Summary

A stand-alone section of two to three pages that should:

1. Briefly describe the intervention (the project(s) that was evaluated.
2. Explain the purpose and objectives of the evaluation, including the audience for the evaluation and the intended uses
3. Describe key aspect of the evaluation approach and methods.
4. Summary of the key findings, conclusions, and recommendations.

Introduction

This section will;

1. Provide a brief explanation of why the evaluation was conducted, why the intervention is being evaluated at this point in time, and why it addressed the questions it did.
2. Identify the primary audience or users of the evaluation, what they wanted to learn from the evaluation and why and how they are expected to use the evaluation results.
3. Identify the intervention (the project(s) that was evaluated
4. Acquaint the reader with the structure and contents of the report and how the information contained in the report will meet the purposes of the mid-term evaluation and satisfy the information needs of the report’s intended users.

Description of the Intervention

This section will provide the basis for report users to understand the logic and assess the merits of the mid-term evaluation methodology and understand the applicability of the evaluation results. The description needs to provide sufficient detail for the report user to derive meaning from the evaluation. In particular, the section will;

1. § Describe what is being evaluated, who seeks to benefit, and the problem or issue it seeks to address.
2. § Explain the expected results map or results framework, implementation strategies, and the key assumptions underlying the strategy.
3. § Link the intervention to the durable solution framework
4. § Identify any significant changes (plans, strategies, logical frame-works) that have occurred overtime and explain the implications of those changes for the evaluation
5. § Identify and describe the key partners involved in the implementation and their roles.
6. § Describe the scale of the intervention, such as the number of components (e.g., phases of a project) and the size of the target population for each component.
7. § Indicate the total resources, including human resources and budgets.
8. § Describe the context of the social, political, economic, and institutional factors, and the geographical landscape within which the intervention operates and explain the effects (challenges and opportunities) those factors present for its implementation and outcomes.
9. § Point out design weaknesses (e.g., intervention logic) or other implementation constraints (e.g., resource limitations).

Evaluation Scope and Objectives

This section of the report will provide an explanation of the evaluation’s scope, primary objectives and main questions.

1. § Evaluation scope-define the parameters of the evaluation, for example, the time period, the segments of the target population included, the geographic area included, and which components, outputs or outcomes were and were not assessed.
2. § Evaluation objectives-spell out the types of decisions evaluation users will make, the issues they will need to consider in making those decisions, and what the evaluation will need to achieve to contribute to those decisions.
3. § Evaluation criteria-define the evaluation criteria or performance standards used. The report should explain the rationale for selecting the particular criteria used in the evaluation.
4. § Evaluation questions- the evaluation questions will define the information that the mid-term evaluation will generate. The report will detail the main evaluation questions addressed by the evaluation and explain how the answers to these questions address the information needs of users.Evaluation Approach and Methods

This section will describe in detail the selected methodological approaches, methods and analysis; the rationale for their selection; and how, within the constraints of time and money, the approaches and methods employed yielded data that helped answer the evaluation questions and achieved the evaluation purposes. The description will help the report users judge the merits of the methods used in the mid-term evaluation and the credibility of the findings, conclusions and recommendations. The description of methodology will include discussion of each of the following:

1. § Data sources-sources of information (documents reviewed and stakeholders), the rationale for their selection and how the information obtained addressed the evaluation questions.
2. § Sample and sampling frame-the sample size and characteristics; the sample selection criteria, the process for selecting the sample (e.g. random, purposive); and the extent to which the sample is representative of the entire target population, including discussion of the limitations of the sample for generalizing results.
3. § Data collection procedures and instruments-methods or procedures used to collect data, including discussion of data collection instruments (e.g., interview protocols), their appropriateness for the data source and evidence of their reliability and validity.
4. § *Performance standards***-**standard or measure that will be used to evaluate performance relative to the evaluation questions (e.g., national or regional indicators, rating scales).
5. § Stakeholder engagement-stakeholders’ engagement in the evaluation and how the level of involvement contributed to the credibility of the evaluation and the results.
6. § Major limitations of the methodology-major limitations of the methodology shall be identified and openly discussed as to their implications for evaluation, as well as steps taken to mitigate those limitations.
7. § Data analysis-procedures used to analyse the data collected to answer the evaluation questions. This will detail the various steps and stages of analysis that will be carried out, including the steps to confirm the accuracy of data and the results. The report will discuss the appropriateness of the analysis to the evaluation questions. Potential weaknesses in the data analysis and gaps or limitations of the data should be discussed, including their possible influence on the way findings may be interpreted and conclusions drawn.

Findings and Conclusions

This section will present the evaluation findings based on the analysis and conclusions drawn from the findings. In particular,

Findings: This section will present findings as statements of fact that are based on analysis of the data. The evaluation findings will be structured around the evaluation criteria and questions so that report users can readily make the connection between what was asked and what was found. Variances between planned and actual results will be explained, as well as factors affecting the achievement of intended results. The assumptions or risks in the project design that subsequently affected implementation will also be discussed.

Conclusions: This section will be comprehensive and balanced and highlight the strengths, weaknesses and outcomes of the intervention. The conclusion section will be substantiated by the evidence and logically connected to the evaluation findings. The conclusion will also respond to key evaluation questions and provide insights into the identification of and/or solutions to important problems or issues pertinent to the decision-making.

Recommendations: The mid-term evaluation will seek to provide very practical, feasible recommendations directed to the intended users of the report about what actions to take or decisions to make. The recommendations will be specifically supported by the evidence and linked to the findings and conclusions around key questions addressed by the evaluation. This shall also address sustainability of the initiative and comment on the adequacy of the project exit strategy.

Lessons Learned

The report will include discussion of lessons learned from the evaluation, that is; new knowledge gained from the particular circumstances (intervention, context outcomes, even about the evaluation methods) that are applicable to a similar context. Concise lessons based on specific evidence presented in the report will be presented in the mid-term evaluation report.

Report Annexes

The Annex section will include the following to provide the report reader with supplemental background and methodological details that enhance the credibility of the report.

1. § ToR for the evaluation
2. § Additional methodology-related documentation, such as the evaluation matrix and data collection instruments (questionnaires, interview guides, observation protocols, etc.) as appropriate
3. § List of individuals or groups interviewed or consulted and sites visited
4. § List of supporting documents reviewed
5. § Project results map or results framework
6. § Summary tables of findings, such as tables displaying progress towards outputs, targets, and goals relative to established indicators.

8. Time frame

The overall evaluation process is expected to take 45 days including preparation, data collection, and analysis and final reporting. The Consultant should be able to undertake some of the tasks concurrently to fit within the planned time-frame, without compromising the quality expected. The assignment is expected to commence on 25^th December 2019, with the final evaluation report expected by 31st January 2020.

9. Authority and Responsibility

WV Somalia will establish an evaluation team to oversee all the related tasks. The DME Manager will be responsible for the overall coordination of all the evaluation tasks with the Consultant. In addition, WVS DME Coordinator/Officers, WASH & Infrastructures Manager, WASH Officers, Quality Assurance & Strategy Manager and WV Germany Monitoring and Evaluation Advisor will provide all the necessary technical and operational support required throughout the evaluation process.

Support from WV Somalia

WV Somalia will be responsible for the following:

1. § Recruit the external Consultant and finalize the consultancy agreement
2. § Share all necessary documents to the Consultant to finalize the evaluation methodology and data collection tools
3. § Provide input for evaluation study methodology, data collection tools and report.
4. § Ensure that input from Project team is circulated and shared with external Consultant
5. Vehicle hire to support the evaluation exercise
6. Working space for the Consultant while in Somalia
7. Recruitment and payment of enumerators
8. Stationery for data collection

§ Overall accountability of the evaluation process
§ Guidance and coordination throughout all the phases of evaluation, keeping communication with external Consultant throughout all phases
§ Provide support to the evaluation technical lead (external Consultant) for the evaluation field visits processes such as orientation and training of enumerators, FGDs and KIIs
§ Closely follow up the data collection process, ensuring quality control, daily debriefing, meeting the timelines set for interview completion;
§ Inform evaluation audience for their involvement in the study and help in setting specific dates for the evaluation field schedule.
§ Provide smartphones/tablets, Kobo Toolbox server for data collection where required.

The Consultant will be responsible for the following:

1. § Review all relevant documents for evaluation study
2. § Develop evaluation study design which includes survey methodology and the data collection tools (review the existing household questionnaire; focus group guides, interview protocol, data entry templates). Besides, prepare a field manual for training, in consultation with evaluation team, reflecting WV Somalia feedback on the methodology. These should be heavily based on the tools used at baseline, midterm to make appropriate comparisons over the life of the project
3. § Designing the xml forms, data entry template, procedures and systems, and training of entry clerks in the use of the template,
4. § Develop the field work schedule in consultation with evaluation team
5. § Conduct training for enumerators during field visits phase, finalize the evaluation schedule
6. § Supervise the data collection process, provide advice and ensure the quality of the data
7. § Conduct interviews (KII) with the WVS and line ministries
8. § Data analysis and report writing. It is expected that at least 2 drafts be provided to WV Somalia with feedback addressed in each round before submission of the final report
9. § Provide required data that is complete and labelled in English (variables and values) for both the SPSS and Microsoft Excel file formats.
10. § Provide final versions of data collection tools.
11. § Provide daily field briefing to the DME Manager, DM&E Coordinator/ WASH Officers, WASH & Infrastructures Manager on the progress and any challenges from the field.

10. Limitations

Time and security may be a major limitation with regard to assessment processes in fragile and versatile contexts such as Somalia and this makes it often challenging to keep up strictly with a set agenda. In addition, In Somalia; households spent a better part of the afternoon hours in prayers and it will be hard for the enumerators to administer many questionnaires per day (in an effort to complete the assessment timely). To address this issue, firstly WV Somalia will allocate extra overflow days for field data collection. WV Somalia team will also work closely with the security department to ensure that the evaluation field processes are conducted in the most appropriate time and secure conditions. Therefore, the Consultant should be able to demonstrate some level of flexibility when required.

11. Documents

The key documents to be reviewed for the evaluation study are as follows:

1. § Project document (proposal, log frame)
2. § Baseline Report
3. § Quarterly, semi-annual and annual and monitoring reports
4. § Training reports
5. § Success stories
6. § Any district level secondary data and other relevant documents and reports.

12. Qualifications of the Consultant

We are looking for a Consultant with the following skills and qualifications;

• § The team leader MUST possess atleast a Master’s Degree in any of the following fields; International Development, Social Sciences, Statistics, Community Development, Development Studies, Local Government or any related field
• § Strong and documented experience in conducting participatory qualitative assessments related food security, livelihoods and water and sanitation programming.
• § Demonstrated experience in leading at least three similar project evaluation studies such as surveys and group interviews,
• § At least 10 years’ experience in conducting baseline and evaluations for complex projects such as livelihoods, education and protection, infrastructure development, health, water and sanitation and hygiene being implemented by non-governmental and private sector actors.
• § A solid understanding of remote learning and use of mobile technology in data collection,
• § Demonstrated experience in leading teams, training local staff in quantitative and qualitative data collection tools including entry template
• § Demonstrated experience in designing survey methodology, data collection tools, processing and analysis of data.
• § Ability to interact with host government, partners as requested by WV Somalia;
• § Strong organizational, analytical and reporting skills, presentation skills, attention to detail, ability to meet deadlines, and proficiency in SPSS or other statistical packages, Microsoft Office and qualitative data analysis software/tools.
• § Previous experience in a fragile country with tight security context will be preferred.
• Capacity to use mobile data collection and GIS tools for data collection, and analysis of survey results.
• § Excellent verbal and written communication in English required.