Terms of Reference for Consultancy Services-
End of project evaluation for the Maternal Infant and Young Child Nutrition (MIYCN) Project in 11 woredas in Afar region - SURVEY COORDINATOR
Save the Children has been working in Ethiopia since 1989 delivering humanitarian and development interventions, and operational in Afar region since 2005 supporting the regional government on different humanitarian interventions and development initiatives. Major thematic areas of experience are livelihoods and food security, WASH, health, nutrition, education, DRR, and early warning in response to drought, flooding and other disasters.
Since May 2015, Save the Children in partnership with UNICEF has been implementing a Maternal Infant Young Child Nutrition (MIYCN) project in 11 woredas of Afar region namely Awra, Dewe, Semurobi, Hadelila, Mille, Adaar, Br’mudaitu, Gulina, Ewa, Telalak and Dalifege. The project continues until December 2017 and aims to contribute to the reduction of undernutrition prevalence among under-five children through improved maternal, infant and young child nutrition. Major expected results are:
· Increased access to MIYCN related services of better quality among Pregnant and Lactating Women (PLW)/caregivers through use of multiple sectoral platforms and community-based contact points.
· Adopting innovative learning agenda with robust monitoring and evaluation to inform decisions for improved programming and scaled-up intervention.
The major activities of the project towards achieving increased access to and quality of MIYCN related services for PLW/caregivers include;
§ Conducting baseline and end-line surveys in programme woredas of Afar
§ Supporting proper recording and reporting of nutrition data and
§ Carrying out formative studies and disseminating study findings and project experiences.
Prior to starting implementation of this project, Save the Children undertook a baseline survey to establish baseline values for optimal infant feeding behavioural indicators including maternal feeding during pregnancy and lactation and complementary feeding of children in the above-listed 11 woredas of the region. A formative research was also undertaken to identify the barriers and motivators that contribute to poor IYCF practices (identified during baseline) in order to develop appropriate behaviour change strategies. An endline IYCF survey will be undertaken to evaluate the extent to which key IYCF behaviours and practices have changed.
A consultant is sought to undertake the final evaluation which will only evaluate the project based on the OECD-DAC criteria – relevance, effectiveness, efficiency, impact and sustainability. The evaluation results will inform the extent to which change has happened in the intervention woredas as well as provide input into similar existing projects or make recommendations for future interventions.
Save the Children will engage an external evaluator (Consultant) in undertaking the final project evaluation for the MIYCN project.
The overall objective of the evaluation is to establish, document and validate the extent to which results have been achieved in the two and a half years of implementation and the resultant impact this has had on the target beneficiaries. It will also assess the level of community and other stakeholders including regional, woreda and zonal officials’ participation and ownership of the implementation process. It shall identify the intended and unintended outcomes, best practices, lessons learned as well as challenges arising from programme implementation. In addition, the evaluation will come up with conclusions and recommendations on the way forward.
The end‐term evaluation will:
1. Generate information on the level of achievement of the intervention objectives, outcomes and wherever possible any evidence of emerging impact.
2. Inform and guide the design or replication of similar existing and future Save the Children projects.
Specifically, the evaluation seeks to address the following issues and provide specific, actionable and practical recommendations for future programming:
I. Assess the project objectives and proposed outcomes by measuring performance against each performance outcome indicator under each result area. Analyze key determinants that, positively or negatively, were critical for obtaining these results (Refer to project logframe available in the annex). The evaluation will be informed by the results of the endline IYCF survey.
II. Assess the relevance of the project in relation to priorities and needs of both the communities and woreda and zonal officials.
III. Evaluate the effectiveness of the project with regards to achieving the stated objectives.
IV. Evaluate the efficiency of the organizational set‐up for the project and systems used in the delivery of the project and to what extent these contributed to or inhibited the delivery of the project outcomes.
V. Assess the level of sustainability (financial, institutional and social) of the individual project components, and identify critical areas that may affect sustainability as well as make recommendations of how sustainability might be improved in future similar projects.
VI. Assess the capacity of the health, education, nutrition and social protection/food security sectors at Woreda and Kebele level in providing nutrition counselling and referral services.
VII. Assess the capacity of communities, community structures and other Civil Society Organization partners to address MIYCN and improve over-all nutrition status of targeted communities.
Some of the key questions for which answers are needed are:
· Did the objectives and activities of the project address priority needs of the target population? Are there gaps in services, in terms of availability and/or quality that still need to be filled?
- What are the major lessons from the program design?
· Was the program beneficiary targeting appropriate in the given context? Are there other beneficiary groups in the area that were not targeted by the project, who have similar priority needs?
· Are the interventions consistent with the needs of the target group? How relevant are they from the point of view of the target group?
· Are the interventions well aligned with existing national and regional development policies and administrative systems of the government? Is it consistent with a policy of supporting ownership?
· Were the proposed interventions technically adequate to address the identified problem of inadequate MIYCN in the target community? To what extent did the project address the identified problem?
· Do proposed innovations around multisectoral platforms and multiple contact points for MIYCN have a potential for replication?
· Was the project consistent and complementary with other activities supported by SCI and other organizations working on the same issue in the area?
· How could the level of participation of different stakeholders have been increased and/or strengthened?
· To what extent do development/humanitarian changes in the target woredas of Afar agree with the planned outputs, results, purpose and goal of the project?
· What are the reasons for the achievement or non-achievement of objectives?
· To what extent, was the programme able to engage leaders, male parents, youth, schools, agriculture extension workers, health extension workers, other ministries at zonal and woreda level in the programme?
· To what extent were different multi-sectoral platforms and contact points utilized and how effective were they?
· To what extent was the capacity building of health workers, teachers, agriculture and food security personnel effective in achieving the objectives of the program?
· What measures have been taken during planning and implementation to ensure that resources are efficiently used?
· How efficiently was the work carried out in terms of time and money? What mechanisms were used to control costs and how effective were these? How could costs be better managed?
· What were the limitations and challenges faced by the project staff? What were the general strength and weakness of the project?
· To what extent was the technical support provided and required to deliver the objectives available during project implementation?
· How many (monthly or quarterly) review meetings were held with the government counterparts and community where staff discussed the project? Were activities altered in light of suggestions shared during those meetings? Share any examples of such modifications?
· What mechanisms were put in place to ensure community accountability?
· How were complaints/feedback to and from the community and other stakeholders handled?
· Is the intervention consistent with the community priorities and effective demand? Is it supported by regional and woreda officials and is it well integrated with local social and cultural conditions?
· Are relevant government institutions involved and to what extent can they continue after the project ends?
· What benefits from the project are likely to be maintained at both government and community levels and which benefits will not be maintained?
· To what extent will the mother to mother support groups and the school nutrition clubs continue after Save the Children’s support is discontinued?
· To what degree have the projects outcomes been achieved at endline in comparison to the baseline survey? Note: determine change in indicators (baseline vs. endline)
· How has the change (intended and un-intended) impacted key stakeholders at all levels?
· To what extent can identified changes be attributed to the intervention? What would have occurred without the intervention?
· What are the factors that cause social behaviour change used in the MIYCN project that can be replicated and/or scaled up after completion of the project?
· What were the innovative approaches/practices devised by the project staff and communities during project implementation? How did we learn from program experience and share our experiences to improve the program and with others?
· Has the information generated through this project been communicated to project stakeholders and the wider nutrition sector?
· How effectively were project lessons (good and bad practice) identified, documented and used for project improvement?
· Explain how problems and opportunities that arose during implementation were acted upon and addressed both by Save the Children and government counterparts?
· How did the program adapt to the drought crisis and how would similar projects in this context be designed to take into account the climatic realities?
The consultant will be expected to develop an evaluation methodology in consultation with Save the Children (technical and MEAL teams). The consultant will be expected to write an inception report following a review of the literature and gaps where he/she will explicitly propose an evaluation methodology, defining the sample size and how selection bias will be avoided. As a minimum, the evaluation process will include the following key steps:
· Review of relevant literature related to the project (c.f. list of reference materials)
· Application of appropriate data collection tools (e.g. questionnaire, checklist etc.) for interviews and discussions with project beneficiaries and stakeholders including woreda and zonal officials and Save the Children staff.
Following are the key stakeholders that need to be consulted during the evaluation:
§ Save the Children staffs (the woreda based nutrition officers, MIYCN advisor, project manager)
§ Woreda and zonal nutrition, health, education and PSNP focal persons
§ Health extension workers
· Agriculture extension workers
· Teachers and school nutrition club members
· Members of the mother to mother support groups
§ community leaders
The following general steps are proposed for conducting the project evaluation:
§ Review the project proposal, Logical Framework, project implementation plans and the M&E plan, and other related project documents like baseline and end line survey report, formative research report ,progress reports and donor reports, case studies and success stories, budget variance reports (BVAs) etc.
Intended Users of the Report:
§ Save the Children (Afar, Addis Ababa, SCUK)
§ UNICEF and DFID (Donors)
§ Regional and Woreda Health office
§ Communities in the implementation woreda’s
The following deliverables are expected:
§ An inception report [covering what sampling strategy the evaluator intends to follow, data collection tools (possibly as annexes) any interview or FGD guides, survey forms, or other data collection instruments to be used, clearly stating the limitations of the proposed data collection methods and the sampling strategy, detailed work plan with level of efforts for each of the proposed team member. Elements of Ethics involving children].
§ Evaluation matrix: which should show how the evaluators plan to answer each of the evaluation questions reflecting the data collection methods set out to answer the questions. It should be presented as an annex with the inception report.
§ A draft report for the study using the format given below in the TORs.
§ A final report after incorporating comments from the implementation team and advisors in digital copy in MS Word (page size A4). A hard copy printed and bound (excluding spiral one) with the pages containing graphs, figures and pictures printed in colour.
§ A PowerPoint presentation containing methodology, key findings, lessons learned and recommendations
The evaluation findings will be presented at regional and woreda level during the closeout workshops. The findings will also be presented to the relevant sectors within SCI at Afar and Addis levels. The executive summary should be provided to the implementers at woreda level as well as to the HEWs and agriculture extension workers and others as deemed necessary.
Final Evaluation report format should use the SCI report template and have as minimum included:
§ Standard cover sheet having the following contents;
o Title of the Study with geographic area, Month and year of execution.
o SCI, UNICEF, DFID and Afar Regional Government Logos.
o Name and address of the evaluation team who conducted the evaluation.
§ Contents of Study will include:
o Acknowledgment
o Executive Summary (background Objectives, Methods, results & conclusion)
o 1. Background [description of project and activities]
o 2. Literature review
o 3. Methodology (having below mentioned components)
§ a. Objectives of the study
§ b. Operational definitions
§ c. Study design
§ d. Sampling Method and Sample Size
§ e. Data Collection Procedure
§ f. Data Analysis
§ g. Significance / Importance of the study, ethical considerations
§ h. Limitations
o 5. Findings should be presented by using OECD-DAC criteria as mentioned above. A section on the endline survey findings compared with the baseline should be presented.
o 6. Conclusions must include a discussion on findings in light of recommended literature review
o 7. Recommendations should be based on the findings of the evaluation that would inform SCI about the areas that need improvement in case of implementing similar nature of projects in future.
o 8. References
o 9. Annexes [management response to the findings, TOR, data collection tools, case studies.]
o Following lists must be given at the start;
§ 1. Table of contents (automatically generated)
§ 2. List of Figures
§ 3. List of Tables
§ 4. Acronyms and abbreviations
The consultant must possess demonstrated knowledge and experience in conducting evaluations using robust research methods. They also must have demonstrated experience of conducting surveys in the health and nutrition sectors.
November 2017
Phase | Activities | Timeline | Date to start | Date to finish |
Phase-1 Desk Review Inception Report | Review of background documents (process documents, project proposals, baseline results, log-frames, strategic plans) and consultation with relevant SC staff | 3 days |
|
|
Design of assessment methodology and tools for data collection | 2 days |
|
| |
Presentation of Inception Report | 1 day |
|
| |
Phase-II Field data Collection | Field work (training of research assistants, data collection and data entry etc.) | (7 – 12 days) |
|
|
Phase-III Data Analysis and Report writing | Draft preliminary report with recommendations
| 7 days |
|
|
Finalize report (following review/vetting) and presentation) | 7 days |
|
|
The Evaluation team must follow Ethical Principles for involving human subjects in a research and obtain written/ verbal consent from the human subjects. Permission from elders/parents must be sought if the children under 18 years are involved as subjects. Signed informed consent of each child and his/ her parents need to be taken after explaining purpose of the study and its usage. Individual responses should not be shared to observe confidentiality of data. SCI has to ensure that no risk or harm to the people involved in this study. In case, the report contains photos; the evaluation team has to take written consent from the photographed person, using the SCI consent forms.
The Evaluation will have a value of 60% for technical fitness and 40% for financial proposal submitted.
Rating Criteria | Score | ||
1 | Understanding of the TOR (5%) |
| |
a | It is clear from the application that the TOR and tasks are clearly understood by the applicant |
| |
2 | Experience of the Firm in related work [supported with clear documentation] (12) |
| |
a | Has done at least five similar consulting work with international NGOs and UN agencies over the past (4 Points) |
| |
b | Has the thematic excellence /expertise in MIYCN (4) |
| |
c | Has managed large and complex undertakings (large studies in terms of sample size, geographic coverage , complexity of methodology, total cost) (4) |
| |
3 | Professional team composition accounts. (8 Points) |
| |
a | Principal investigator has strong evaluation capacity to manage MIYCN evaluation; has done similar evaluation (4 Points) |
| |
b | Composition of the team (number and qualification vis a vis the approach and methodology proposed /the requirement indicated in the TOR) (4) |
| |
4 | Methodology proposed to undertake the survey( 30 Points) |
| |
a | Demonstrates Clear Understanding of Tasks and combination of proposed methods is excellent (5) |
| |
b | Presents clear logical work plan that shows of how proposed activities of the consultancy will meet the consultancy objectives (10). |
| |
c |
| ||
d | Proposes clear logistics plan: survey team distribution and vehicles to transport survey teams, duplication of questionnaire (5) |
| |
e | proposes clear data quality control methods /measures (5) |
| |
f | proposes clear data management and processing plan (5) |
| |
5 | Time frame-schedule. (5 Points) |
| |
a | Can complete task within proposed timeframe /within reasonable time period |
|
The bench mark for selection criteria is that opening the financial is if a firm fulfilled 45% and above out of 60% great on technical evaluation.
The consultant must possess demonstrated knowledge and experience in conducting evaluations using robust research methods. They also must have demonstrated experience of conducting surveys in the health and nutrition sectors.