Evaluation

Evaluation can be defined as the systematic and objective examination of the humanitarian sanitation response to determine the worth or significance of an activity or programme. It is intended to draw lessons to improve the response and enhance Accountability.

Key Actions

• Share evaluations in an appropriate format with all key stakeholders so that the findings can be discussed and applied, e.g. through workshops, reports, presentations and communityCommunityA group of people sharing something in common. meetings.
• Include all partners and other actors when developing the monitoringMonitoringMeasures progress and checks whether a programme or intervention is working according to plan. It is the planned, systematic and continuous checking of a sanitation intervention to ensure it is doing what was intended, that allocated funds are being used effectively, that… and evaluationEvaluationThe systematic and objective examination of a humanitarian intervention to determine the worth or significance of an activity, policy or programme and intended to draw lessons to improve policy and practice and enhance accountability. The key evaluation criteria are relevance,… framework and, where possible, carry out joint evaluations.
• Budget for an evaluationEvaluationThe systematic and objective examination of a humanitarian intervention to determine the worth or significance of an activity, policy or programme and intended to draw lessons to improve policy and practice and enhance accountability. The key evaluation criteria are relevance,… in the sanitation programme. Calculate costs such as the evaluators, interpreters, logistics (e.g. transport and accommodation) and dissemination (e.g. printing, communityCommunityA group of people sharing something in common. meetings and workshops).
• Clarify the purpose of the evaluationEvaluationThe systematic and objective examination of a humanitarian intervention to determine the worth or significance of an activity, policy or programme and intended to draw lessons to improve policy and practice and enhance accountability. The key evaluation criteria are relevance,…, the type of information needed and develop a Terms of Reference with a timeline and budget.
• Develop a Logframe with indicatorsIndicatorsIndicators are the ‘signals’ that enable measurement of progress and objectives and therefore of change. to enable an evaluationEvaluationThe systematic and objective examination of a humanitarian intervention to determine the worth or significance of an activity, policy or programme and intended to draw lessons to improve policy and practice and enhance accountability. The key evaluation criteria are relevance,… of the inputs (resources used), activities (what was done), outputs (what was delivered), outcomes (what was achieved) and impact (long-term changes). Develop indicatorsIndicatorsIndicators are the ‘signals’ that enable measurement of progress and objectives and therefore of change. which are disaggregated by age, gender and disability.
• Match the evaluationEvaluationThe systematic and objective examination of a humanitarian intervention to determine the worth or significance of an activity, policy or programme and intended to draw lessons to improve policy and practice and enhance accountability. The key evaluation criteria are relevance,… methods to the requirements of the evaluationEvaluationThe systematic and objective examination of a humanitarian intervention to determine the worth or significance of an activity, policy or programme and intended to draw lessons to improve policy and practice and enhance accountability. The key evaluation criteria are relevance,… and be accessible to and inclusive of marginalised groups. EvaluationEvaluationThe systematic and objective examination of a humanitarian intervention to determine the worth or significance of an activity, policy or programme and intended to draw lessons to improve policy and practice and enhance accountability. The key evaluation criteria are relevance,… methods may include: Key Informant Interviews, Observation and Transect Walks, Pocket Chart Voting, questionnaire-based Surveys and Community Mapping.
• Collect qualitative and quantitative dataQuantitative DataInformation related to quantities. It is numerical and asks how many, how much, or how often. from different sources (triangulationTriangulationCompares several different data sources and methods to cross check and confirm findings, helping to reduce bias.), analyse it using appropriate methods and compile the findings into a report.
• Avoid the common pitfalls of evaluations, including: (1) Focusing on easy to reach geographic areas, (2) not collecting baselineBaselineDetermines the starting point for subsequent monitoring. The term ‘baseline’ can refer to the situation before the emergency or provide initial data to compare with an ‘endline’ survey. (‘before intervening’) data, (3) not respecting data protection and or putting participants at risk, e.g.in insecure areas, (4) neglecting consultation with less visible groups, e.g. women, older people and persons with disabilities, (5) ignoring seasonal or geographical WASH differences, (6) collecting too much or unnecessary information, (7) focusing the evaluationEvaluationThe systematic and objective examination of a humanitarian intervention to determine the worth or significance of an activity, policy or programme and intended to draw lessons to improve policy and practice and enhance accountability. The key evaluation criteria are relevance,… merely on outputs, not considering outcomes, behaviour change and impact, (8) not widely sharing the results, so the information is lost and not used to adapt programming, (9) not informing the target group about the results of the evaluation.EvaluationThe systematic and objective examination of a humanitarian intervention to determine the worth or significance of an activity, policy or programme and intended to draw lessons to improve policy and practice and enhance accountability. The key evaluation criteria are relevance,…

Relevance/Importance

There are numerous reasons for undertaking evaluations including to review innovations, gain evidence, demonstrate success or challenges as part of a learning process, assess value for money and to be accountable to key stakeholders such as donors and, especially, to the affected population.  An evaluationEvaluationThe systematic and objective examination of a humanitarian intervention to determine the worth or significance of an activity, policy or programme and intended to draw lessons to improve policy and practice and enhance accountability. The key evaluation criteria are relevance,… looks at the overall changes which can be attributed to a sanitation programme and examines the outcomes achieved, the relevance, efficiency, sustainability and wider impact on people’s lives. It can produce recommendations to improve the programme (including capacity strengthening if needed) and capture learning to inform future policy and practice. It is an important aspect of Accountability. Sharing and using evaluationEvaluationThe systematic and objective examination of a humanitarian intervention to determine the worth or significance of an activity, policy or programme and intended to draw lessons to improve policy and practice and enhance accountability. The key evaluation criteria are relevance,… findings encourages transparency and Learning in the sector.

Overview

There are different types of evaluations depending on the objectives. Some evaluations are carried out at, or after, the end of the programme (or mid-way in longer-term programmes) and aim to provide accountabilityAccountabilityThe obligation to demonstrate that work has been conducted in compliance with agreed rules and standards. Sphere describes accountability as the process of using power responsibly, taking account of and being held accountable by different stakeholders and primarily those who are… and influence future policy and practice. Real-time evaluations are carried out during the programme, are interactive and involve multiple stakeholders; the evaluator acts as a facilitator to generate an overview of the programme and provide immediate feedback so that issues can be addressed during the response. All types of evaluationEvaluationThe systematic and objective examination of a humanitarian intervention to determine the worth or significance of an activity, policy or programme and intended to draw lessons to improve policy and practice and enhance accountability. The key evaluation criteria are relevance,… can be external and independent or conducted by an agency with the support of an external evaluator or by staff members.

It may be appropriate to do joint evaluations in collaboration with other programme staff, partners and other organisations (e.g. within the WASH Cluster) to minimise duplication of resources. Some evaluations have a strong focus on accountabilityAccountabilityThe obligation to demonstrate that work has been conducted in compliance with agreed rules and standards. Sphere describes accountability as the process of using power responsibly, taking account of and being held accountable by different stakeholders and primarily those who are… to the affected population, empowering them to play a key role in carrying out and contributing to the process in order to strengthen ownership of the programme and ensure that they are in a position to make use of the findings.

Key evaluationEvaluationThe systematic and objective examination of a humanitarian intervention to determine the worth or significance of an activity, policy or programme and intended to draw lessons to improve policy and practice and enhance accountability. The key evaluation criteria are relevance,… criteria include:

Relevance: asks whether the programme is doing the right things, e.g. is the sanitation programme meeting the needs according to the context? Does the programme target the right people in terms of geographical areas as well as vulnerabilities to WASH-related health risks?

Effectiveness: analyses whether the programme has achieved its objectives and intended results and examines the factors influencing the achievement of those objectives, e.g. has the sanitation programme achieved its objectives of providing accessible and inclusive toilets and safe faecal sludgeSludgeMixture of solids and liquids, containing mostly excreta and water, in combination with sand, grit, metals, trash and/or various chemical compounds. A distinction can be made between faecal sludge and wastewater sludge. Faecal sludge comes from on-site sanitation technologies,… management services? To what extent can these changes be attributed to the programme? If intended results did not occur – why not?

Efficiency: measures both quantitative and qualitative outputs in relation to the inputs, e.g. how efficient is the installed treatment technology at reducing pathogens or the organic load in faecal sludgeSludgeMixture of solids and liquids, containing mostly excreta and water, in combination with sand, grit, metals, trash and/or various chemical compounds. A distinction can be made between faecal sludge and wastewater sludge. Faecal sludge comes from on-site sanitation technologies,…? Does this method make the best use of the resources available? Were there alternative options to improve sanitation conditions?

Impact: examines whether there were significant or lasting changes resulting from the programme and whether they were intended or unintended, positive or negative, e.g. has the goal of the programme been achieved? Have there been any changes in public health? Has the programme made a real difference to the affected population?

Sustainability: evaluates the extent to which the net benefits of the intervention will continue or are likely to continue, e.g. have people been supported to continue using, maintaining and repairing the sanitation facilities? What behaviours have changed as a result of the intervention and how likely are these changes to last? Has local capacity strengthened?

Coherence: considers how well the intervention fits with existing country plans and local priorities (e.g. does the programme align with Government policies) as well as with programmes and interventions of other response agencies?

ParticipationParticipationAims to empower people and involve them in decisions that affect their lives. Inclusive planning and participation are key ingredients to achieve sustainability in humanitarian interventions. They exceed consultation which involves seeking input from stakeholders. The different…: examines the level of active community engagementCommunity EngagementConnects the community and other stakeholders so that people affected by crisis can participate and have more control over the response and its impact on them. and inclusive involvement of all segments of the affected population in planning, management and decision making of the programme in order to achieve appropriate ownership over the outcomes. It includes aspects such as people’s motivation, capacities to engage and opportunities they are given to participate.

Existing National Standards, Sphere Standards, the Core Humanitarian Standard and the Code of Conduct can be used as references to assess the quality of the programme in conjunction with the programme objectives and indicatorsIndicatorsIndicators are the ‘signals’ that enable measurement of progress and objectives and therefore of change..