Evaluation / Impact Assessment of improved access to clean water activities for drought affected communities in Afar, Ethiopia
Background of the evaluation
The Protestant Agency for Diakonia and Development (PADD) is an international development and emergency aid agency registered in Ethiopia as international NGO with the Federal Republic of Ethiopia Charities and Societies Agency, registration number 1295. The Regional Office (RO) works with partners in Ethiopia, Somalia, South Sudan and Sudan. The main goal of the engagement of the Protestant Agency for Diakonia and Development in Ethiopia is to contribute to the improvement of food security/livelihood. In addition, support is given to local partners intervening in the areas of awareness raisin, non-formal education/skill building, health and WASH.
PADD provides humanitarian aid worldwide through Diakonie Katastrophenhilfe (DKH), the global emergency department. DKH supports people who have fallen victim of natural disasters, war and displacement and who are not able to cope on their own in the emergency situation they find themselves in.
DKH has implemented several projects on improving water access to Afar communities with its Ethiopian Partner organisation APDA (Afar Pastoralist Development Association) since 2010. The projects include the component “increased access to clean water” (through the construction/maintenance of traditional local reservoirs Birikuts and ponds). For attribution purposes, it is important to mention that the partner organization has been already working on these activities before 2010. Areas of implementation of the DKH supported projects were in various Woredas of Afar Region - Ethiopia.
Evaluation of the Projects:
K-ETH-1105-0002 | APDA (Afar Pastoralist Development Association) | Promotion of drought resilience in the Afar Region | 01.05.2011 | 31.08.2013 |
K-ETH-1106-0003 | APDA (Afar Pastoralist Development Association) | Afar drought response initiative targeting women, girls, men and boys of the Afar pastoral community | 01.06.2011 | 30.11.2011 |
K-ETH-1209-0005 | APDA (Afar Pastoralist Development Association) | Water, Hygiene, Sanitation and Capacity Building Interventions for Drought and Acute Watery Diarrhoea | 15.09.2012 | 15.05.2013 |
Commissioned by: Diakonie Katastrophenhilfe
Purpose, objectives and users of evaluation
The purpose of this evaluation is to provide DKH with evidence on the longer-term impact of the projects implemented, especially in regard of water infrastructure put in place and access, utilisation, and changes experienced by project beneficiaries and other affected groups especially after the projects’ intervention periods. This will allow for the replication of successful or promising practices and strategies and avoidance of negative or harmful effects that the intervention may have had. Overall, the findings will inform future programming in drought prone pastoralist areas.
Specific objectives, scope, and users of the evaluation
The specific objectives of the evaluation are:
Scope of the evaluation
The evaluation is limited to the programmatic success or failure of the projects defined above. Management has to include the following elements:
Contextual analysis including:
Database of water infrastructure incl. year of construction, physical status, feasibility of rehabilitation if necessary, water capacity, user numbers (human m/f and animals), water quality, water retention capacity (months of dry-spell covered), average duration of water supply and water outage during a year, comparison with respective water demand for each water infrastructure assessed (calculation based on population figures and Sphere standards for human consumption/sharing with animals if applicable)
OECD-DAC evaluation criteria for humanitarian action (based on ALNAP Evaluation in Humanitarian Action; see also the annex for further detail):
The reference frame to answer these evaluation criteria is extends from the initial project duration until the present in the perspective of a longer-term aggregate ex-post cross-examination of all projects in light of the evidence today.
Specific additional research questions are:
Users of the evaluation
The evaluation will be shared with DKH Regional Office (Nairobi) and HQ staff (Berlin) as well as the implementing Partner organisation of the projects under evaluation. It will also be shared with the PADD Regional Office in Ethiopia. The evaluation will also be shared with prospective implementation Partners in projects related to water access in drought-prone pastoralist areas. Further sharing of the results is at the discretion of the DKH Regional Office Nairobi.
Evaluation Design and Methodology
The consultant will be required to develop an inception report (incl. evaluation matrix), which outlines design and methodological requirements and proposes tools to use and plan of action in carrying out the assignment. The evaluation matrix should contain as a minimum the complete set of evaluation questions/sub-questions, indicators, data sources, analytical method applied to each question. The inception report together with these ToR will form the basis for a discussion and finalisation of the concrete assignment between DKH and the consultant before the start of the assignment. The inception report should also include draft indicators for impact measurement.
The evaluation design should deliver disaggregated data and short analyses for each project defined above plus a comparison and aggregate evaluation across all the projects.
The methodology must include a sound mix of secondary and primary data collection and analysis, as well as quantitative and qualitative methods, and deliver gender, age, and environment conscious differentiation. Care needs to be taken to analyse the attribution of change to the specific DKH supported projects. The sampling strategy should be clearly defined and justified.
For contextual and health related data in particular, secondary data and desk research are suitable. Primary data is expected with regards to the other aspects of the assignment. Technical aspects should be covered by quantitative data collection. Utilisation, preferences, and water quality and hygiene aspects require a mix of quantitative and qualitative approaches as does the evaluation as per the OECD-DAC criteria.
All primary data collection shall be documented in an electronic database which will be shared together with the evaluation report. Data privacy and protection standards apply. As such, data on water infrastructure and corresponding target groups needs to be geo-referenced. Data/results from Focus Group Discussions (FGD) need to be documented in word or excel. Key Informant Interviews and HH questionnaires should be recorded either with electronic data collection methods based on ODK or Kobotoolbox or otherwise be documented electronically in a suitable format.
Data analysis should include visual representation of results: mapping, charts, photo documentation as well as triangulation between quantitative and qualitative elements.
Responsibilities of the Consultant and DKH
Consultant:
DKH:
Annex
OECD-DAC Criteria for Evaluations in the Humanitarian Sector
In 1991, the OECD-DAC, focusing on the most common problems noted in development projects, proposed four quality criteria – relevance, effectiveness, sustainability and impact – and the value criterion of efficiency (OECD-DAC, 1991).
A few years later, it adapted these criteria for EHA in complex emergencies (OECD-DAC, 1999), adding coverage and coherence, suggesting appropriateness as an alternative to relevance and connectedness as an alternative to sustainability, and proposing two new criteria: coordination and protection. These criteria reflected the biggest problems seen in humanitarian action in the 1990s.
Understanding the different OECD-DAC criteria:
Relevance
Did we plan the right thing? Do we do the right thing? To what extent are our objectives, planned activities and planned outputs consistent with the intended outcome and impact? Are there differences between the time when the project was planned and designed and today?
Efficiency
Impact
Sustainability
Connectedness
Coverage
Coherence
Coordination
Note:
It is important to use the DAC criteria intelligently rather than mechanistically. DKH expects consultants to identify and cross-check a relevant number of evaluation questions with DKH, that are realistic to address, taking into account the context, data availability, and the scope and resources available to the evaluation.
Evaluation Ethics:
The evaluation process shows sensitivity to ethnic groups, gender, beliefs, manners and customs of all stakeholders and is undertaken with integrity and honesty. The rights and welfare of participants in the evaluation are protected. Anonymity and confidentiality of individual informants should be protected when requested and/or as required by law.
Evaluation team members should have the opportunity to dissociate themselves from particular judgements and recommendations. Any unresolved differences of opinion within the team should be acknowledged in the report.
Stakeholders are given the opportunity to comment on findings, conclusions, recommendations and lessons learned. The evaluation report reflects these comments and acknowledges any substantive disagreements. In disputes about facts that can be verified, the evaluators should investigate and change the draft where necessary. In the case of opinion or interpretation, stakeholders` comments should be reproduced verbatim, such as in an annex, to the extent that this does not conflict with the rights and welfare of participants.
Water infrastructure includes: Birikuts, Ponds, Water Pans and Rehabilitated Water Pumps / Water Points. Include assessment if the SODIS method for water purification would be relevant to target groups and if it is already known or could be promoted: http://www.sodis.ch/methode/anwendung/ausbildungsmaterial/index_EN
Profile, Expertise, and Coordination of the evaluation team
The lead consultant for this assignment should meet the following criteria:
Process, reporting and timetable of evaluation
Tasks and Time requirements
No | Duties and Responsibilities | Consultancy Days | Timeframe |
1 | Preparation: Analysis of relevant documents Developing design and instruments of the evaluation Preparation of an inception report | 2 |
|
2 | Hand-over inception report to contractor |
|
|
3 | Implementation of the Evaluation in the field for example: Conducting of observations (GIS) Electronic HH survey KII on various levels Focus Group Discussions (female / male disaggregated) | 9 |
|
4 | Data Management and Analysis Cleaning and documentation of data Analysis of data Drawing inferences/recommendations Development of Draft Evaluation Report | 5 |
|
6 | Presentation and discussion of draft report with DKH | 1 |
|
7 | Final reporting and handing over of final report incl. raw database, annexes and data collection instruments developed | 1 |
|
| Total number of days | 18 |
|
| of this: traveling days | 2 |
|
Outputs and Reporting requirements
The consultant provides a concise de-briefing report of max. 3 pages upon return from the field. After analysing the data thoroughly and drafting the final report, the draft report will be presented to DKH Regional Office and feedback will be given. The final report will incorporate this feedback.
Structure of the Evaluation Report (English):
Meta-information (cover page)
Key data of the evaluation: Title, number, duration of the project evaluated, title of the evaluation, principal of the evaluation (who commissioned the evaluation), contractor of the evaluation, date of the report
Executive Summary:
A tightly drafted, to-the-point, free-standing document, describing the key issues of the evaluation, main analytical points, conclusions, lessons learnt and recommendations (max. 2-3 pages).
Table of Contents
Introduction
Purpose of the evaluation, scope and key questions. Short description of the project, relevant socio economic and political frame conditions
Evaluation design/methodology
Key results: Observations / Findings / Conclusions
Annexes
Including ToR, database established, schedules, travel documentation, data collection tools, list of stakeholders, sampling strategy, list of interviewees, listing and description of FGDs, electronic formats of the surveys and GIS database, etc.
Observations, Findings, Conclusions and Recommendations have to be based and explicitly linked in the report to evidence. The analytical link between Findings, Conclusions, and Recommendations needs to be transparent and sound (e.g. by providing a tabular overview or a clear narrative).
The report has to be submitted in English. It should not be longer than 30 pages excluding annexes. All annexes should be presented in English language as well as much as possible.