Terms of Reference
Baseline Study – USAID Somalia’s Accelerated Education Activity
Summary
Through the present Terms of Reference/TOR, CARE Somalia seeks to procure consultancy services to conduct a baseline study for the USAID Somalia’s Accelerated Education Activity (AEA). The baseline study will enable the use of a quasi-experimental evaluation at the subsequent evaluation round.[1] The subsequent evaluation study will be contracted directly by USAID Somalia. The baseline study will use a complexity-aware, gender-responsive, inclusive, and conflict-sensitive approach for tool design, data collection protocols, and data analysis.
The selected consultancy firm will be responsive for data collection and analysis/ reporting services, working in close collaboration with USAID Somalia, CARE, and USAID Somalia’s Evaluation, Learning and Monitoring Initiative (ELMI)[2] to design tools, data collection protocols, and the analysis framework for this study. This ToR outlines the purpose, methodology, requirements, timeline, and deliverables for the baseline study consultancy, anticipated to take place in October-January 2024.
Background
The USAID Somalia’s Accelerated Education Activity (AEA) is a $30M, five-year initiative (2024-2029) designed to assist the Federal Ministry of Education, Culture and Higher Education (MoECHE) to strengthen and expand the provision of accelerated basic education (ABE) and non-formal education (NFE) in country and facilitate the transition of ABE and NFE graduates into further education and livelihoods.
The USAID Somalia’s AEA will support the MoECHE’s priority objective of expanding access to basic education while also enhancing equity and completion rates. The 2022-2026 Education Sector Strategic Plan (ESSP) identifies the enhanced access to ABE as one of the key approaches to achieve this priority objective, seeking to expand ABE enrolment to 250,000 students by 2026, out of whom 50% female.[3] To do so, the ESSP highlights the need to expand platforms for ABE delivery; strengthen the quality of ABE provision, achieving 72% completion by 2026; and provide appropriate teaching and learning materials to ABE classes.[4] The ESSP also prioritizes the need to expand the provision of Adult Education (NFE), reaching 75,000 students by 2026, thus requiring the development of associated teaching and learning materials and teacher training.[5] The USAID Somalia’s AEA will contribute to these priorities by enabling 67,072 out-of-school children and youth to enroll or remain in ABE/ NFE, equipping them with the foundational skills necessary to transition into further education and/or livelihood opportunities. Additionally, AEA will support the MoECHE and Federal Member States Ministries of Education (FMS MOEs) to consolidate and expand the provision of ABE and NFE, strengthening regulatory systems and quality assurance processes.
The USAID Somalia’s AEA will target areas where an estimated 1.9M children and youth ages 10-19 are in need of ABE, and nearly 300,000 youth in need of Adult Education/ NFE services.[6] Those include the Banadir Regional Administration (BRA) and Hirshabelle, Jubaland, and South West states of Somalia.
Theory of Change
The Theory of Change (ToC) for the USAID Somalia AEA considers that:
If there is increased equitable access to safe, free, relevant, accelerated basic education and non-formal education programs (Intermediate Result/IR 1);
If there are improved learning outcomes in foundational skills areas (literacy, numeracy and social-emotional learning) and other key skills as appropriate (IR 2);
If there is increased community engagement and local ownership of ABE and NFE programming (IR 3); and
If there is strengthened capacity of government at local and federal levels to oversee the provision of quality public education (IR 4), then
There will be a greater number and a more equitable distribution of out-of-school children and youth (OOSCY) with the foundational skills necessary to transition to further learning and/or social and economic opportunities within their communities (expected outcome). This outcome will ultimately contribute to the achievement of USAID Somalia’s Development Objective/DO 2, strengthening the social capital of marginalized groups and the human capital of women, thus enabling marginalized Somalis to more effectively withstand shocks and stresses.
A graphic representation of USAID Somalia’s AEA Theory of Change is provided below.
Baseline Study Objectives
The baseline study aims to:
-Develop a baseline for applicable indicators, enabling future assessment of progress vis a vis targets at the subsequent evaluation round.
-Identify factors affecting outcomes and map those against the Theory of Change, validating it.
-Create in-depth profiles of schools, students, and teachers, identifying vulnerabilities and needs.
-Generate evidence to inform the implementation of other ABE/NFE-focused interventions in country.
The baseline findings will be used to:
-Refine targets for applicable indicators.
-Inform adaptations to the Theory of Change and Activity design/modalities of implementation.
-Inform policy design and changes.
Evaluation Questions
-What are the critical factors influencing the acquisition of literacy and numeracy skills among ABE/ NFE students? How do those map out against AEA’s proposed interventions?
-Are there differences in learning outcomes by sex, geography, disability status, language, displacement status, and minority condition? What are the recommendations for adaptations to reduce learning gaps?
-To what extent are ABE/NFE teachers adopting improved practices aligned with Universal Design for Learning principles? Which subgroups of ABE/NFE teachers are most/less likely to apply those?
-To what extent are ABE/NFE teachers adopting gender-responsive and inclusive classroom practices?
Qualitative sub-question: What are the attitudes and perceptions of ABE/NFE teachers towards female students, students with disabilities, displaced students, and those with a minority background?
-What are the actual attendance rates among ABE/NFE students and teachers? Which factors are affecting student attendance?
-What factors are affecting student retention?
-To what extent are CECs implementing their expected tasks in ABE/NFE centers, including enrolment, attendance tracking, teacher monitoring, safeguarding, school maintenance, and fundraising?
Qualitative sub-question: What are the main barriers to CEC performance?
-What school/system-level structures are in place to facilitate transition from ABE into formal education? What are the critical gaps?
-How and to what extent are state, regional, and district-level structures supporting the implementation of ABE and NFE courses? What are the perceived capacity/ resource gaps for ABE/NFE support?
Methodology
The evaluation will use a two-arm quasi-experimental design (QED) with a mixed-method approach. The QED will enable the following comparisons:
1. Learning and retention trends compared between ABE L1/L3 and primary school students:[7]The evaluation will compare learning and retention trends between longitudinally tracked ABE levels 1 and 3 students and cross-sectional samples[8] of primary school students in grades 1 and 5 at the baseline, respectively. Both samples will be drawn from the same locations or neighboring locations in the case of ABE-only learning centers. At the final evaluation, the learning trends for longitudinally tracked ABE students (L1 at the baseline) will be compared to those of a cross-sectional sample of grade 8 graduates. To ensure comparability, the external evaluator will refine samples using propensity score matching (PSM). A schematic representation of the comparisons over time is presented in the figure below.
Figure 2: Evaluation design – ABE x primary education comparisons for learning outcomes
Therefore, this consultancy will require the external evaluator to randomly select samples of ABE L1 and ABE L3 students as well as students in primary grades 1 and 5.
2. Learning and transition trends comparison between NFE students and OOSCY: The evaluation will compare learning and transition trends for longitudinally tracked NFE students with cross-sectional, randomly selected samples of OOSCY in the same age ranges at the baseline (15-25 years) and subsequent evaluation round in sampled school communities. The external evaluator will identify randomly selected NFE and OOSCY samples at the baseline, using propensity score matching to ensure comparability and thus avoiding disparities in terms of gender, education history, displacement history, and motivation for enrolment.
Figure 3: Schematic representation of the evaluation design to assess differences in learning and transition trends between NFE students and comparison groups of youth drawn from targeted school communities.
Sampling. At the baseline, the external evaluator will randomly select a sample of 84 schools among the 293 schools targeted in Year 2 in the three states and the BRA. The total number of sampled schools per state will be allocated proportionally to the distribution of the ABE/NFE student population in each state and the BRA. The sampled schools will be longitudinally tracked at the subsequent evaluation round.
At the baseline, the external evaluator will draw six samples – ABE L1 students, ABE L3 students, primary school students in grades 1 and 5, NFE students, and OOSCY ages 15-25. Each sample will include 703 individuals, or an average of nine students per class and nine OOSCY in the same catchment area.[9] The sample size is calculated considering 95% confidence level; 80% power; an effect size of 0.25; a design effect of 2; disaggregation by sex in each sample; and an attrition rate of 40%. Therefore, the total sample will include a total of 1,406 ABE students (disaggregated by L1 and L3); 703 NFE students; 1,406 primary school students (disaggregated by grades 1 and 5 – ABE comparison group); and 703 OOSCY ages 15-25 (NFE comparison group). The samples for ABE and NFE students will be pre-selected by the external evaluator from AEA’s student database, which includes profiles for each student.
Tools
The baseline will use both quantitative and qualitative tools. Quantitative tools will include the following:
1. Student assessment, formed by:
- Learning assessmentsbased on the Early Grade Reading Assessment (EGRA) and Early Grade Mathematics Assessment (EGMA)[10]. The learning assessments will include expanded tasks catering to the expected learning levels in ABE L3-4 and primary grades 5 and 8, thus avoiding a ceiling effect. The learning assessments will be co-developed by the evaluator in consultation with CARE and the MOECHE and piloted in a sample of 10 non-assessed schools with students in grades 4, 6, and 8. The learning assessment tasks (words, stories, writing tasks, math problems) will be in Af-Mahatiri (standard Somali), with instructions provided in the student’s mother tongue.
- Working memory test: Pictorial test used to assess the students’ capacity to purposively retain information for a short period of time. Working memory is a key cognitive capacity for reading comprehension and can be severely affected by malnutrition. The working memory test was previously used in Somalia by another initiative (SOMGEP-T).
- Youth Leadership Index: Scale used to assess self-perceptions of leadership. The tool was previously used in Somalia by other initiatives (SOMGEP-T, AGES, GEEPS).
- Adolescent/youth survey, including items on the student’s socio-economic background;[11] motivation for enrolment in ABE/ NFE/ primary; dietary diversity and food consumption; education experience; teacher attendance; teaching practices; perceptions about gender and inclusion in class; future expectations for transition; safety and security; and access to water, sanitation, and menstrual hygiene supplies (girls only) at school.
2. School survey, including:
- Head teacher questionnaire/ observations: Observations of school condition (infrastructure, hygiene and sanitation practices[12]); provision of meals; school management; types of support received; recordkeeping; monitoring support.
- Headcounts: On-site verification of teacher and student attendance.
3**. Classroom observations**: Will assess the use of improved classroom management and literacy teaching practices; gendered practices; inclusive practices; use of traditional / negative practices; availability of formative assessments; availability and use of textbooks; and availability and use of reading material. The classroom observation tool will use the format previously used by AGES, incorporating additional questions on reading practices.
The baseline will use the following qualitative tools:
Tool
Respondent
Areas of Inquiry
Vignette exercises
ABE and NFE students
Gender and social norms
Time allocation / attendance
Perceptions about education
Aspirations for transition and perceptions of ability to transition into livelihoods
Relationships with peers/ teachers
Vignette exercises
ABE teachers
NFE teachers
Gender and social norms
Time burden
Safeguarding
Perceptions about transition into higher education levels/ livelihoods
Perceptions of technical support and professional development
Risk mapping
ABE/NFE students
Safety and security
Safety perceptions for different subgroups
Focus Group Discussions/ FGDs
CEC members
Gender and social norms
Implementation of the School Improvement Plan
Safeguarding
Perceptions of technical support
Fundraising
Key Informant Interviews/ KIIs
FMS MOEs coaches/ District Education Officers (DEOs)
Gender and social norms
Perceptions of ABE/NFE
Perceptions of technical support
Safeguarding
The qualitative tools will be applied in a subsample of eight schools proportionally distributed across states to replicate the overall distribution of AEA schools.
Ethical approval
CARE will seek formal approval from the MoECHE and FMS MoEs to conduct this study. However, the selected consultancy firm should be able to use its own institutional review board (IRB) or seek approval from an external IRB for this study, in line with the requirements outlined in USAID’s Policy Brief on Ethics in Research and Evaluation in the Education Sector.
Selection and training of enumerators
The external evaluator is responsible for the recruitment, training, and oversight of enumerators. The external evaluator will be required to assess enumerators during training to ensure adequate understanding of data collection protocols, ethics, and gender and inclusion mainstreaming in data collection processes. The external enumerator will provide a five-day training to ensure adequate performance, including mock practices and a field practicum at school. All enumerators will be required to complete safeguarding training and sign CARE’s Safeguarding Policy prior to conducting any field activities.
At a minimum, the enumerator training should include:
- AEA’s design and purpose of the evaluation
- Safeguarding and fraud prevention (sessions provided by CARE staff)
- Informed consent, assent, confidentiality, and ethics (including practical examples)
- Gender and inclusion in data collection
- Electronic data collection
- Overview of quantitative tools
- Mock practice
- Qualitative data collection
- Data quality assurance
- Practicum in schools/ communities
- Protocols for daily reporting / data transfer
Quality assurance
The evaluator is responsible for quality assurance at all levels. During data collection, team leaders will be responsible for on-site verification of data collection procedures and guidance to enumerators. In parallel, the external evaluator will be required to verify daily data uploads. Any suspected cases of fraud, either by enumerators or by schools, will be reported immediately to CARE in line with the Fraud Prevention Policy. The external evaluator will be required to provide AEA with weekly data collection reports. CARE’s Director of Research will also cross-verify the datasets for quality assurance.
The external evaluator is required to outline quality assurance procedures in the proposal submitted to CARE. The quality assurance procedures should respond to the needs in a context where at least part of the schools will only be accessible to local teams due to security concerns.
Analysis
The analysis of baseline data will be guided by the evaluation questions listed above. In addition, AEA will require the external evaluator to include the following:
-Profile of ABE/NFE students
-Baseline results for relevant outcome indicators
-Learning results by subtask and performance bands
-Learning results disaggregated by sex and other subgroups of interest (language, minority status, disability, displacement, pastoralist, farming background, etc)
-Analysis of factors affecting reading, numeracy, attendance, and retention (as per evaluation questions above) and how those factors map out against AEA’s design, thus assessing the likelihood of influencing outcomes
-Triangulation of qualitative findings with learning, attendance, and retention results
-Contributions to the learning agenda (CARE will share the AEA learning agenda with the selected evaluator)
-Recommendations for adaptations to AEA’s design and implementation.
Timeline
The draft baseline study report should be submitted no later than December 10, 2024. The final baseline study report should be submitted no later than January 9, 2025.
Key Deliverables
This consultancy requires the completion of the following deliverables:
- Inception report: Includes a summary of the baseline methodology, tools, sampling approach, quality assurance procedures, and proposed analysis framework.
- Enumerator training report.
- Data collection report.
- Clean quantitative datasets with complete codebooks.
- Complete transcripts and translations of qualitative data.
- Draft baseline study report.
- Presentation summarizing key baseline findings.
- Final baseline study report, incorporating feedback from USAID, MOECHE, and AEA.
Required qualifications
Bidders should meet the following criteria:
- Key personnel with expertise and demonstrated experience in advanced quantitative analysis, use of mixed methods, and education evaluation.
- Demonstrated ability to collect quality data in South Somalia, including in rural and urban areas.
- Previous experience in conducting donor-funded evaluations of similar scope and size.
Bidding process
External evaluators interested in submitting a bid should submit technical and financial proposals. The technical proposal should include, at a minimum:
-An outline of the proposed methodology.
-Risk analysis
-Approach to incorporating gender and inclusion in the baseline data collection and analysis.
-Key qualifications and past experience.
-Samples of work of similar scope and complexity, conducted in Somalia or in similar contexts.
-References.
-CVs of key personnel. The replacement of key personnel will require prior approval from CARE.
– The financial proposal should include all the costs related to the baseline including but not limited to personnel costs, travel related costs (flight and vehicle) for all the participants involved, payments to the enumerators/research assistants, training of the enumerators/research assistants etc
– Consultants are required to provide their certificate of registration from the countries they are applying from.
– Questions about TOR should be sent to [email protected] not later than October 19, 2024.
The technical and financial proposals are due no later than October 24, 2024, through the email below
[1] Which will be contracted directly by USAID Somalia.
[2] ELMI will be responsible for the final evaluation of AEA, and will therefore be consulted on the design of the baseline study.
[3] MoECHE, National Education Sector Strategic Plan 2022-2026, pg.65
[4] Ibid
[5] Ibid, pgs.73-74
[6] CARE (2024) Initial Needs Assessment – Somalia Accelerated Education Activity, pgs.9-10
[7] This comparison will respond to the following evaluation questions at the final evaluation:
-Are there differences in the learning curves for ABE and primary education students? How effective is ABE, compared to formal primary, for students with similar socio-economic backgrounds?
-How efficient is ABE in terms of retention, compared to formal primary school, for students with similar socio-economic backgrounds?
[8] Since ABE is a compressed course, a comparison with a longitudinally tracked group of primary school students would be biased – i.e. a L1 ABE student assessed at the baseline, who will have reached the equivalent of grade 5 at midterm, cannot be compared with a longitudinally tracked primary school student at grade 1 during the baseline, who would have only reached grade 3. Therefore, such comparison requires the use of cross-sectional primary school student samples.
[9] The average is 8.3 students/ class to meet a total sample of 703 individuals/ class. With local class size variations, it is anticipated that 8-9 students will be assessed to meet the sample size.
[10] The math assessment will include a financial literacy test, building upon an existing tool previously used in Somalia by SOMGEP-T.
[11] IDP status; deprivation of basic needs; disability status; language; displacement status; marital status; if the respondent has children; if the respondent is in labor; with whom the student lives.
[12] Availability of water and soap at latrines; condition of latrines; use of latrines by students and by girls only
The technical and financial proposals are due no later than October 24, 2024, through the email below
– Questions about TOR should be sent to [email protected] not later than October 19, 2024.
More Information
- Job City Mogadishu
