2. Definition of evaluation and case study

EVALUATION

 

Evaluation of the project is an accountability function which determines whether the project’s predefined outputs have been achieved in a timely manner, and whether they are efficient, effective and have an impact on the overall objective of the project. It is often carried out at project completion and often also assesses the impacts of the project on customers, clients and stakeholders. More specifically, evaluations of programmes can give insights into:

  • whether the needs of your project are effectively addressed;
  • the outcomes your project has achieved;
  • the alignment and relationships between your activities and the outcomes;
  • the people who were involved in your project;
  • the definition of success for your project;
  • the outputs and immediate results compared to what you expected.

 (Adapted from Zarinpoush, 2006)

 

 

 

Evaluations can furthermore identify the strengths and weaknesses to help improve a programme (see box 1.1 below).

Box 1.1

“In one country, an evaluation was conducted on a programme designed to train and provide capital to female micro- entrepreneurs. The programme was part of a broader strategy aimed at fostering women’s empowerment through better income and livelihood opportunities. The evaluation found that the intended results were achieved: The training and micro- enterprise programmes were successful and women who participated in the programme experienced an increase in self- employment and income. Moreover, the women felt more empowered to make decisions for themselves and within their households. However, the evaluation also found that many of the women were unhappy at the end of the programme, as there had been an increase in marital and partner problems and a few relationships had ended as a result of the changes in the women’s empowerment. Some of the women reported that their partners were not prepared for these changes and did not know how to relate to them. They suggested that maybe these problems could have been avoided had there been some counselling provided to their partners at the beginning and during the programme to better prepare them for the coming changes.” (UNDP, 2009:52)

 

 

An evaluation report should ask questions that address the following criteria:

-          Relevance (assess the relevance of the project design with the needs and issues that stakeholders or water resources are facing)

-          Effectiveness (assess the extent to which planned activities and their outcomes contribute to the overall project objectives)

-          Efficiency (carry out a cost-benefit or cost-effectiveness analysis to outline the benefits and the costs of a programme)

-          Sustainability (assess how the outcomes can be sustained after project completion)

 

Many types of evaluation tools exist; each should be used in accordance with your needs and the subject of your evaluation. A non-exhaustive set of evaluation tools is presented below:

  • Survey - a set of predetermined question about certain topics that are answered by a target audience (quantitative).
  • Interview - a set of questions about certain topics that are posed to a target audience and followed by additional questions and conversations (qualitative).
  • Focus Group - group discussion about certain questions (qualitative).
  • Evaluation Form - set of questions that determine the participant’s opinions, attitudes and understanding once a project activity is complete (quantitative).
  • Site visits - combination of observation and interviews that occur in the project’s environment (qualitative).
  • Documentation - administrative record of project activities (e.g. reports, minutes of meetings, registration forms, national statistics, etc.) (quantitative and qualitative).
  • Anecdotes - Stories and narratives about an event, experience or an individual related to the project (qualitative).

Evaluation Case study: The Lor Township Water Supply Project and the Water Project Toolkit checklist.

The following fictional case study for a water sector project is presented to illustrate in practical terms what has been explained in section “1.2 Evaluation”, giving

 

-        the description of the project,

-        the Logical Framework, which specifies how progress will be evaluated,

-        the project-related evaluation report example measured against the Water Project

-        Toolkit checklists (chapter 10) in order to control whether the evaluation report contains the necessary information.

 

Project description:

General Objectives:  To promote good health and economic development in the State of Lor in Nissa[1] through the provision of adequate water supply and sanitation services


Specific objectives: To raise the demand level from about 35 litres/capita/day in 2000 to 106 litres/capita/day for an estimated population of about 400 000 by the year 2010. To improve and increase the supply of drinkable water in order to meet the needs of the town of Lor and its surroundings.

 

Beneficiary of the project: Lor State Water Board

Executing Agency: Lor State Water Board

Duration: 10 years

Estimated cost: x euro.

Final Project cost: X euro

Expected results:

1.       Construction of a new treatment plant of 455 000 m3/day capacity,

2.      Construction of a high-lift pumping station,

3.       Construction of a stand-by power station,

4.       79 km of pumping mains and transmission and distribution pipelines and provision logistics,

5.      Provide technical assistance for institutional building, engineering and construction services.

Logical Framework:

 

Description and intervention

Indicators (specify what to measure- in qualitative/quantitative terms)

Source of information (where you can get the necessary information)

Overall Objective

Promote good health and economic development in the State of Lor in Nissa through the provision of adequate water supply and sanitation services

 

- Service level by population served

 

- Statistics for incidences of water-related diseases in Lor since 2000 and incidences of outbreaks of epidemics

 

National reports on water and sanitation coverage.

Statistics of the Ministry of Health regarding incidence and trends of water-borne diseases or epidemics.

Results of Governmental or NGO surveys

 

Purpose( specific objective)

Raise the demand level from about 35 litres/capita/day in 2000 to 106 litres/capita/day for an estimated population of about 400 000 by the year 2010

- Demand level liter/capita/day of 2000 and 2010

- Lor State Water Board’s operational and commercial statistics

- Project Completion report

1st result-Activities

Alleviate the shortage of drinking water required for domestic, social and economic uses by improving the water treatment facility and the reticulation systems

 

- Capacity of the water treatment facility to increase by 45 500 m3/day.

- Lor State Water Board’s operational and commercial statistics

- Supervision reports.

2nd Result- Activities

 

Etc……..

 

  •  
  •  

 

The Water Project Toolkit suggests checklists in order to make sure that the Evaluation Report contains the necessary information. These can be used to help build your evaluation tool by suggesting what should be considered in the evaluation of a project.  Based on the above mentioned case study, the following table shows how to use the Water Project Toolkit checklist and what should be done to improve the impact of the project. 

Evaluation main findings

Water Project Toolkit checklist (chapter 10)

Relevance: Satisfactory

 

 

 

 

The project goal of improving the health of people was relevant to the country’s overall macro-economic policy but the achievement was only partial as the benefits did not fully reach the poor.

 

 

 

 

Sector issues and institutional arrangements were not fully dealt with at the time of project formulation.

 

 

 

Project objectives were in line with water sector goal but lack integrated approach.

 

 

The project should have been formulated and implemented ensuring effective stakeholders involvement and participation. Therefore examine:

  • If groups were prevented from participating, determine the causes.
  • Analyse the actual and potential involvement of NGOs, users, private sector and all the other stakeholders. If their involvement was ineffectual, suggest means to strengthen their performance and participation in future projects.
  • Identify local knowledge and local institutions that contributed or could have contributed to the project.

 

 

 

Provision should be made for effective inter-agency and inter-sector planning. If this was weak, lessons should be learnt for the future. Therefore examine:

  • If the institutions or their internal structures were inappropriate, identify alternative partners or make recommendations for future institutional reform / capacity building.
  • Investigate the reasons for inadequate support and make recommendations for change.
  • Where agencies failed to perform adequately, identify the weaknesses and recommend necessary capacity building measures

 

p.193 Institutional and management Principles

 

Effectiveness: Unsatisfactory

Physical:

 

Implementation of the plant construction process was satisfactory; however some original components were not implemented, such as the  replacement of 50 km of distribution lines and the provision of technical assistance for institutional development.

 

 

 

Poverty alleviation and gender:

 

Women have benefited from the project, and the attendance to school of children, particularly girls, improved considerably in areas served by the project (Ministry of Health statistics). However the service’s coverage of the urban poor is inadequate. Provision of consumer connections in high density areas lagged behind those provided to the better-off sections of the community.

  ! No baseline data or indicators were set at the programming and identification stage to really evaluate the poverty alleviation and gender impact!

 

 

Environment:

The additional water supply generated additional wastewater which was not addressed. Lack of proper sanitation and drainage affected some areas and helped spread disease.

 

 

Transfer of technology:

Little has been achieved; qualified staff are still made up of only a  few top-level officials.  Disbursements by components were not strictly observed and funds allocated for technical assistance were used to cover cost overrun in the physical component of the project.

Ownership by stakeholders and user groups of services provided by the project is essential in ensuring sustainability. Therefore examine:

  • Assess the comprehensiveness and utility of the stakeholder analysis conducted during formulation and recommend improvements.
  • Determine if the findings of gender analysis influenced project design and implementation.
  • If the intended and stakeholders have failed to take on ownership, assess the reasons and recommend changes for future projects

Evaluation must determine to what extent intended social development has been achieved and what unexpected impacts may have occurred. Therefore examine:

  • Where data are unavailable or social impact indicators are inadequate, re-consider the monitoring exercise. If necessary, use proxy indicators and interviews.
  • Use Social Impact Analysis (see part 3) as part of project evaluation and to make recommendations for future projects.
  • If the number of beneficiaries, the extent of benefits, or the equity of their distribution are less than projected, identify the causes.
  • Evaluate the effectiveness of any targeting policy and make recommendations.

Environmental damage may result because insufficient time and money is invested in collection and analysis of data during and after. Therefore examine:

  • If data are unavailable or the environmental impact indicators are inadequate discuss with those responsible. If necessary, use proxy indicators.
  • Where significant unforeseen environmental impacts have occurred, identify the reasons; indicate mitigating measures, and ensure that improved environmental analysis is incorporated in future project cycles.
  • Assess the efficacy and acceptability of the mitigating measures introduced.
  • Make recommendations for improved long term monitoring and reporting.

In evaluating the appropriateness of technology and its influence on the wider results of the project, the accuracy of underlying data is critical. Therefore examine:

  • Assess whether all technical design and study assumptions were valid. Where inaccurate assumptions have led to poor design, make appropriate recommendations for future projects.

 

 

p.194 Social Principles

p.195 Environmental Principles

p.196 Technological Principles

Efficiency: Unsatisfactory

 

Inadequate baseline data to calculate the  Economic Internal Rate of Return (EIRR)

 

The expected indirect benefits of the project of spurring the establishment of industries by removing the major constraint of a lack of reticulation system did not materialise.

No baseline data or indicators were set at the programming and identification stage to really evaluate the socio-economic impact of the project!

Sub-optimal revenue collection revenues cannot cover the O&M costs. Most of the consumer water meters that were procured as part of the project had not been installed and therefore it was impossible to recover charges for water supplied.

 

 

Projects must demonstrate economic benefit and financial accountability if they are to be sustained over the long term. Therefore examine:

  • Where data are unavailable or economic impact indicators are inadequate discuss with those responsible for monitoring. If necessary, use proxy indicators.
  • Identify the causes of any significant variation between predicted and actual economic benefit.
  • If there was a major overspend, identify the causes and analyse its effect on calculating economic impact. Ensure that the findings are incorporated in future programming.
  • Assess the disbursement and management capacity of the implementing agencies and plan disbursement schedules for future projects of these agencies accordingly.

 

p.194 Economic and Financial Principles

 

Sustainability:  Unsatisfactory

 

Technical Viability:

Operation of the pumping plant suffered from lack of spare parts arising from a lack of proper assessment of requirements and lead times. 

- Premature rusting of pipes occurred because of a change in specifications of transmission and distribution without adequate safeguards, sometimes resulting in the contamination of treated water and water leakages.

 

 

 

 

Financial Viability:

Although progress has been made to increase water tariffs, the current rates did not cover the long-run marginal cost of supply. Commercial and accounting capabilities of the Lor State Water Board remained extremely weak, its finances were fully stretched and its cash flow was restricted.

 

In evavuating the appropriateness of technology and its influence on the wider results of the project, the accuracy of underlying data is critical. Therefore examine:

  • Assess whether all technical design and study assumptions were valid. Where inaccurate assumptions have led to poor design, make appropriate recommendations for future projects.

 

Technological solutions must be acceptable to the target users and compatible with the environment. Therefore examine:

  • Assess the attitudes of all the key stakeholders to the installed technology and their role in its selection.
  • Evaluate the training delivered during project implementation and the provision made for continuing user training.
  • Review the operation of infrastructure. Assess whether it meets design specifications and the appropriateness of those specifications as can be reflected in the uptake of the users and managers.
  • Discuss any shortfall in implementation with the government agencies involved, particularly where this may have user acceptability or environmental consequences.

 

Sustainability of infrastructure and equipment can only be achieved if the technical and financial requirements for maintenance are met. Therefore examine:

  • Assess systems of cost recovery for O&M, the availability of technical expertise, spares, and the institutional arrangements for providing the resources and labour to carry out maintenance. Make practical recommendations for improvement.

p.196 Technological Principles

Lesson learnt and recommendations

It is important to identify the project’s problems and pitfall but also its successes. Analyse them and identify learning points to avoid future problems and repeat the successes.

 

 

References and further readings:

EC, 2011, Water Project Toolkit: Water resources management for sustainable development. Available online: http://www.aquaknow.net/en/water-project-toolkit

EC, 2012, Specificity of evaluation compared to monitoring and auditing. Available online: http://ec.europa.eu/europeaid/evaluation/methodology/methods/mth_vad_en.htm

UNDP, 2009, Handbook on planning, monitoring and evaluating for development results. Available online: http://web.undp.org/evaluation/handbook/documents/english/pme-handbook.pdf

UN, nd, A Manual on Monitoring and Evaluation for Alternative Development Projects. Available online: http://www.unodc.org/documents/alternative-development/Manual_MonitoringEval.pdf

 

Zarinpoush F., 2006, Project evaluation guide for non-profit organizations: Fundamental Methods and Steps For Conducting Project Evaluation.On behalf of Imagine Canada. Available online: http://library.imaginecanada.ca/files/nonprofitscan/en/csc/projectguide_final.pdf

 

 

 

[1] The country and state’s name were entirely invented for the purpose of the case study.