Consultancy – For Project End Line Evaluation for Roots Ethiopia 97 views0 applications

Terms of Reference (ToR)

For Project End Line Evaluation for Roots Ethiopia

1.  Background

Roots Ethiopia is a 501(c) 3 non-profit organization incorporated under the Wisconsin Statutes in the United States of America. Its principal office is 14 Longmeadow Circle, Madison, Wisconsin, 53717 or such other places in the State of Wisconsin, as the Directors may deem appropriate.  Roots Ethiopia is registered as a Foreign Charity by the Federal Democratic Republic of Ethiopia with Agency for Civil Society Organizations (ACSO), under Registration No. 3847. It works together to improve schools, educate children, and enable livelihoods. These are community priorities, so they’re our priorities.

Roots Ethiopia operates in a wide variety of settings in a large geographic area in Southern Nations, Nationalities and Peoples Regional State (SNNPRS), particularly in 17 selected Woredas (districts) located in Kembata-Tembaro, Hadiya, Wolayta, and Alaba zones with diverse sets of partners and beneficiaries.

In Sep. 2017, in order to respond to critical educational challenges facing rural primary schools of the selected Woredas, the organization has signed an operational agreement with regional Bureau of Education (BOE), Bureau of Finance and Economic Development (BOFED) and Bureau of Women Children and Youth Affairs (WCYA), initially for a period of three years. Over the past three years, it has been implementing three core components aimed at reducing the obstacles that inhibit school children from realizing their educational potentials and development goals. The three project components were learning resource project (LRP), school sponsorship program (SSP) and Self-help groups (SHG).

As a result of this project intervention, 200 students were supported through SSP, 240 women were supported through SHG and more than 36,899 students were benefited from LRP component.

The overall goal of the project is to contribute towards improving access to and quality of primary education in southern region of Ethiopia.

The objectives of the project are:

v  to create improved educational opportunities for individual children from disadvantaged family backgrounds, and increase their educational attainment;

Ø  VIA student sponsorship, school supplies, nutritional and medical support for 200 children each year facing economic hardship to attend school;

Ø  AS MEASURED BY THIS COHORT COMPARED TO ZONE:

§  Number of 1^st grade students completing year by gender

§  Number of 8^th grade graduates continuing to Secondary School

§  Number of 8^th grade students completing 8^th grade year by gender

v  to strengthen the capacity of families to fully engage in economic activities and increase their income that enables them to send their children to school; and the following are specific objectives:

§ VIA the provision of grants and skills training for 320 economically or socially marginalized families to strengthen their livelihoods, and generate sustainable incomes;

§ AS MEASURED BY:

§  Increase in % of family  members engaged in sustainable employment activities (e.g. can afford housing, food resources, and education)

§  Increase in # of family children attending school who are school aged

v  to equip 25 schools’ libraries and classrooms with suitable furniture that encourages learning, contributes to students’ physical and psychological welfare;

Ø  VIA the supply of an average of 700 quality reference books, teaching tools, science lab supplies, sport kits and other instructional materials to positively impact the learning and teaching process;

Ø  VIA the creation of an environment where children with special educational needs in 5 schools are supported, and their individual differences and needs are addressed;

Ø  VIA the establishment of 4 school-based youth sport clubs, enhance the physical and mental development of young people, and help them develop life skills and wellbeing;

Ø  VIA the creation of a supportive work environment for average 100 school teachers and leadership by providing professional skills and refresher trainings and resources;

Ø  VIA the creation of a targeted tutoring program for students preparing for critical tests (e.g. 8^th grade test)

Ø  AS MEASURED BY:

o   Increased percentage of school-aged children from the community enrolled in school, particularly girls

o   Increase in average # of school years completed per student

o   Increased literacy rate in community by age groups and gender

o   Increased percentage of students graduating from the highest grade that the school offers

o   Increased % of attendance

The Project Outputs are:

The outputs resulting from the intervention include:

v  Distribution of essential educational materials to 25 primary schools;

v  Supplying an average of 700 books and setting-up school libraries in selected primary schools;

v  Equipping classrooms of beneficiary schools with adequate furniture;

v  Establishing Special Education classroom in five selected beneficiary schools;

v  Improving school sanitation facilities by building improved 2 latrines;

v  Reproducing and disseminating hygiene promotional materials;

v  Establishing of four school based sport clubs in four zones;

v  Granting school sponsorships for 200 students ;

v  Establishing 12 self-help-groups and providing professional and skills trainings.

The Desired Impact is:

v  Increased % of low income, school-aged children from the community enrolled in school, especially girls

v  Increase in average # of years in school per child enrolled in the program

v  For girls: Increase in average age of marriage for girls vs. average community age for girls.

v  Increase in % of family members engaged in sustainable employment activities (e.g. can afford housing, food resources, and education)

v  Increased literacy rate in community by age groups and gender

v  Increased percentage of students graduating from the highest grade that the school offers

Therefore, RE has decided to hire a consultant to evaluate the implementation of the project RE has been implementing in the last three years (2018-2020) as well as the gap between reality and the desired impact.

2.Purpose of the Evaluation

The purpose of the end line evaluation is to assess the performance of the project and capture project achievements, challenges and best practises to inform future similar programming.  The evaluation will also ensure accountability towards our donors and the beneficiaries of the program. On the other hand, it offers a learning aspect for all stakeholders. The evaluation will also identify key lessons learned, challenges and the flexibility of the programme to adapt and respond to the changes and sustainability of the project.

3.    The end  line evaluation will have the following four objectives:

Objective 1: Evaluate to what extent Roots Ethiopia delivered effective, efficient, relevant and timely activities to beneficiaries as set in the project proposal;

Objective 2: Assess whether the collaboration between Roots Ethiopia and line Bureaus of Education and Women, Culture, Youth and sports, Zone and Woreda education offices have added value to the interventions with a positive effect on beneficiaries and other stakeholders. What has contributed to this added value and what has not?

Objective 3:  Identify and assess key gaps in impact measurement that can be bridged using information already collected and supplied to the Ministry of Education from the Learning Resource Project schools.

Objective 4: Identify and assess key lessons learned, challenges and draw recommendation for future programming of Roots Ethiopia based on MOE and community goals; the end line evaluation should assess the following evaluation/review criteria and include all of the following questions:

Objective 1: Evaluate to what extent Roots Ethiopia delivered effective, efficient, relevant and timely activities to beneficiaries as set in the project proposal;

Effectiveness:

The end line evaluation should assess the following:

v  To what extent have the planned objectives in the project proposal have been reached, per indicator, disaggregated by gender and age?

v  To what extent have the project activities contributed to the overall goal? Was the project effective in increasing educational opportunities and improve their academic achievements?

v   What were the major factors influencing the achievement of the objectives of the project?

v  What opportunities for collaboration have been utilized and how have these contributed to increased effectiveness? Or otherwise?

v  Have proper accountability and risk management framework(s) been in place to minimize risks on program implementation?

 Efficiency: 

v  How efficient was the delivery of project by Roots Ethiopia, not only in terms of expenditure, but also in terms of implementation of activities?

v  Was the project activity implementation (modality) considered to have been cost-efficient, while not compromising quality?

v  What would have been opportunities within project to reach more beneficiaries with the available budget or to reduce costs while reaching at least the same number of beneficiaries without compromising quality?

v  Were alterations made to the program design in terms of collaboration during the implementation phase based on the reality on the ground?

v  What were the outcomes of these choices for effective and efficient program implementation?

 Relevance/Impact:

v  How relevant were the objectives and activities implemented by the project, in addressing the educational challenges of students and poor family needs in targeted communities?

v  How do beneficiaries perceive the relevance of the project and how has the activities implemented improved their lives?

v  Are there any stories of change?

v  How has the collaboration between Roots Ethiopia, government stakeholders and line bureaus contributed to appropriate response of specific needs and priorities of the beneficiaries?

v  To what extent was the project able to adapt and provide appropriate response to context changes and emerging local needs, and the priorities of beneficiaries?

 Timeliness:

v  Were the project design and interventions timely in responding to the needs on the ground?

v  Were the activities timely implemented in comparison to project planning?

v  Were funds available in time during implementation of the activities to respond to new developments?

v  To what extent did the collaboration between Roots Ethiopia and line bureaus and other stakeholders contributed to efficient and timely coordination of logistic activities and processes?

 Reach:

v  To what extent have the project beneficiaries been reached and what mechanisms were in place to improve coverage?

v  What are the main reasons that the project provided or failed to provide to its target beneficiaries proportionate to their need?

 Quality:

The end line evaluation should assess the overall quality of the implementation. It is important to include beneficiaries’ opinion on the quality of the services received.

v  What mechanisms are in place to track project implementation of Roots Ethiopia projects? (i.e. internal monitoring, evaluation, learning and reporting  (MERL) and quality assurance mechanisms)?

v  How have these mechanisms been utilized to increase quality within the project?

v   Did the quality of activities delivered by the project meet the needs and expectations of the beneficiaries?

v   What do beneficiaries feel that could be improved in contribution to improving local capacity?

v  To what extent did the project interventions contributed to build long-term community capacity?

v  To what extent was the project participatory in all the project cycle?

Objective 2: Assess whether the collaboration between Roots Ethiopia, and line Bureaus  of Education and Women, Culture, Youth and sports, Zone and Woreda education offices have added value to the interventions with a positive effect on beneficiaries and other stakeholders. What has contributed to this added value and what has not?

Learning:

v  Which of the interventions, approaches, and modalities/strategies have been most effective according to Roots Ethiopia and line bureaus and other stakeholders;

v  Is there any substantial evidence on how project learning were generated and applied to improve the delivery or effectiveness or efficiency of activities?

v  Who benefited from shared learning experiences (e.g. quarterly, joint field visits, workshops, provision on best approaches and methodology), mainly the NGOs or also the local sector, community members and beneficiaries?

v  How did the different actors learn from these experiences?

Visibility:

What measures have been taken to create visibility of the project’s added value, towards line bureaus and including beneficiaries?

Joint activities:

What joint activities were undertaken during the implementation of project both at a field level as well as at a country office level? (Please provide concrete, short, substantial, cases of evidence).

Objective 3:  Identify and assess key gaps in impact measurement that can be bridged using information already collected and supplied to the Ministry of Education from the Learning Resource Project schools.

Measure Alignment:

v  Which educational success measures are considered critical by the community and the MOE?

v  What type of data is collected regularly by the MOE at the Woreda level about each LRP? Do partner NGO’s (such as Roots) have access to this data?

Objective 4: Identify and assess key lessons learned, challenges and draw recommendation for future programming of Roots Ethiopia based on MOE and community goals.

The end line evaluation should at least include one lesson learned and recommendation per evaluation category, i.e. effectiveness, efficiency, relevance etc.

v  What are the key lessons learned per objective?

v  To what extent has the delivery of response activities contributed to effective, efficient, relevant and timely delivery of outputs and enhanced impact for the beneficiaries?

Sustainability:

v  Will the changes caused by this programme continue beyond the life of the project?

v  What mechanisms have Roots Ethiopia and stakeholders put in place to sustain the key program Outputs and Outcomes?

v  How has the programme worked with local government offices to increase their capacity in a sustainable way?

v  What motivations /mechanisms exist for government stakeholders to continue playing these roles?

v  What are the risks facing sustainability of programme Outputs and Outcomes?

v  What gaps in programming exist based on the goals of the community and the Ministry of Education?

4.    Methodology 

The detail methodology will be developed by the consultant as well as all relevant tools will be presented in the inception report. The three evaluation objectives mentioned above should be assessed including all research questions under each objective.

The data collection should include the use of a number of approaches to gain a deeper understanding of the outcomes of the project including:

·         Desk review of background documents (project document, project monitoring data, progress report, mid-term review report (if any), field visit reports etc.)

·         Key informant interviews with Roots Ethiopia Staff ( both in the field and Addis), key community members/beneficiaries, local stakeholders,  and representatives from the line bureaus to gather substantial anecdotal evidence on the effectiveness, efficiency, relevance and timeliness of the project activities implementation and delivery

·         Focus group discussions (e.g. with stakeholders, and community members, facilitators. The FGD will serve as input for the narrative anecdotal evidence.

·         A learning event in Hossana to disseminate lessons learned through a presentation and a workshop facilitated by the consultant;

Next to the data collection method, an appropriate and strategic sampling method should be selected e.g. snowball sampling, purposeful, random sampling or mixed purposeful sampling methods.

5.    Deliverables:

The consultant is expected to lead, accomplish and submit the following deliverables within the agreed timeframe and budget:

·         An inception report, which will serve as an agreement between parties on how the evaluation will be conducted.

Items to address in the inception report:

·         Understanding of the issues and questions raised in the ToR

·         Data sources; how to assess the questions in the ToR

·         Research methodology, including suggested sample size

·         Schedule of activities and traveling (timeline)

·         Proposal for a learning event/validation of evaluation findings in Hossana

·         Detailed budget

·         Appropriate validated draft data collection tools (e.g. methodological guidelines, group interview questions)

·         Raw data in any of the following statistical packages (STATA, SPSS, cSPro) and also transcribed qualitative scripts

·         A max of 35-page draft and final evaluation report (in MS Office and PDF for final), excluding annexes and in English.

The report should be in the format indicated below to be submitted to Roots Ethiopia Country office. It is preferable to illustrate the results by appropriate graphs, visuals, tables and/or a dashboard with an accompanied explanatory text.

The report should consist of:

a) Executive Summary in bullets (max. 2 pages)

b) Introduction

c) Methodology, including sampling and limitations

d) Analysis and findings of the evaluation. The analysis should be done according to the objectives:  evaluation objective 1; evaluation objective 2; evaluation objective 3, including an implementation strategy for the recommendations

 e) Address concerns, lessons learned and comments from Roots Ethiopia

 f) Stories of change and quotes from respondents

 g) Conclusions for each of the end line evaluation objectives

 h) Recommendations for future projects

 i) Annexes:

§  Relevant maps and photographs of the evaluation areas where necessary

§  Bibliography of consulted secondary sources

§  Finalized data collection tools

§  List of interviewees with accompanying informed consent forms

§  PowerPoint presentation of preliminary findings to Roots Ethiopia

§  Learning event for project stakeholders

7. Selection procedure and requirements

Selection of consulting firms or individual consultant will be based on the following criteria:

A.   Relevant qualification and experience and technical proposal. This will account for 70% of the assessment score. Consulting firms or individual consultants shall be selected based on evidence of their experience in undertaking similar assignments. Having expertise in project/program evaluation and demonstrated experience in educational evaluation is mandatory. Applicants may be assessed based on information/evidence in their application and interview with short-listed candidates to ensure that they possess the required skills and sufficient professional experience.

B.    Technical proposal

The technical proposal should be sound with clear objectives, strategies and relevant methodologies. It should also show the content and relevance of the methodologies and tools that will be developed. The technical proposal should also have an indicative work plan that shows how the allocated working days/ timeline for the different components of the work.

C.   Cost of the assignment (financial proposal/offer). This will account for 30% of the total selection criteria. The cost should be in Birr inclusive of all costs.

The individual consultant or consulting firms are expected to possess equipment such as laptops necessary for implementation of tasks.

8. Place and Duration of the assignment

Consulting firms or individual consultant can work from home when preparing the report but required to go to the project sites in SNNPR- Hadiya, Kembeta-Tembaro, Wolayta and Halaba zones to meet RE staff, students, teachers, project beneficiaries (Self-help groups) and government stakeholders.

The duration of assignment and delivery of the end line evaluation report will be maximum of 30 working days.

9. Conflict of Interest

In order to ensure the impartiality of consulting firms or individual consultants, the consulting firms or consultants should be independent from the institutional structure and staff of Roots Ethiopia Country Office, and other parties involved. Each consultant will be required to sign a declaration of conflict of interest. The selected consulting firm or individual consultant must demonstrate appropriate ethical conduct and must respect the confidentiality of the information and documents to which they will have access to.

10.  Coordination, Reporting and Liaison

The consulting firms or individual consultant/s will be working under the supervision of the Interim Executive Director or the CD with the overall guidance to be provided by RE Senior management team (SMT). Coordination, reporting and liaison will be undertaken through the Interim director or the CD.