Evaluation of the project is an accountability function which determines whether the project’s predefined outputs have been achieved in a timely manner, and whether they are efficient, effective and have an impact on the overall objective of the project. It is often carried out at project completion and often also assesses the impacts of the project on customers, clients and stakeholders. More specifically, evaluations of programmes can give insights into:
(Adapted from Zarinpoush, 2006)
Evaluations can furthermore identify the strengths and weaknesses to help improve a programme (see box 1.1 below).
“In one country, an evaluation was conducted on a programme designed to train and provide capital to female micro- entrepreneurs. The programme was part of a broader strategy aimed at fostering women’s empowerment through better income and livelihood opportunities. The evaluation found that the intended results were achieved: The training and micro- enterprise programmes were successful and women who participated in the programme experienced an increase in self- employment and income. Moreover, the women felt more empowered to make decisions for themselves and within their households. However, the evaluation also found that many of the women were unhappy at the end of the programme, as there had been an increase in marital and partner problems and a few relationships had ended as a result of the changes in the women’s empowerment. Some of the women reported that their partners were not prepared for these changes and did not know how to relate to them. They suggested that maybe these problems could have been avoided had there been some counselling provided to their partners at the beginning and during the programme to better prepare them for the coming changes.” (UNDP, 2009:52)
An evaluation report should ask questions that address the following criteria:
- Relevance (assess the relevance of the project design with the needs and issues that stakeholders or water resources are facing)
- Effectiveness (assess the extent to which planned activities and their outcomes contribute to the overall project objectives)
- Efficiency (carry out a cost-benefit or cost-effectiveness analysis to outline the benefits and the costs of a programme)
- Sustainability (assess how the outcomes can be sustained after project completion)
Many types of evaluation tools exist; each should be used in accordance with your needs and the subject of your evaluation. A non-exhaustive set of evaluation tools is presented below:
The following fictional case study for a water sector project is presented to illustrate in practical terms what has been explained in section “1.2 Evaluation”, giving
- the description of the project,
- the Logical Framework, which specifies how progress will be evaluated,
- the project-related evaluation report example measured against the Water Project
- Toolkit checklists (chapter 10) in order to control whether the evaluation report contains the necessary information.
General Objectives: To promote good health and economic development in the State of Lor in Nissa through the provision of adequate water supply and sanitation services
Beneficiary of the project: Lor State Water Board
Executing Agency: Lor State Water Board
Duration: 10 years
Estimated cost: x euro.
Final Project cost: X euro
1. Construction of a new treatment plant of 455 000 m3/day capacity,
2. Construction of a high-lift pumping station,
3. Construction of a stand-by power station,
4. 79 km of pumping mains and transmission and distribution pipelines and provision logistics,
5. Provide technical assistance for institutional building, engineering and construction services.
Description and intervention
Indicators (specify what to measure- in qualitative/quantitative terms)
Source of information (where you can get the necessary information)
Promote good health and economic development in the State of Lor in Nissa through the provision of adequate water supply and sanitation services
- Service level by population served
- Statistics for incidences of water-related diseases in Lor since 2000 and incidences of outbreaks of epidemics
National reports on water and sanitation coverage.
Statistics of the Ministry of Health regarding incidence and trends of water-borne diseases or epidemics.Results of Governmental or NGO surveys
Purpose( specific objective)
Raise the demand level from about 35 litres/capita/day in 2000 to 106 litres/capita/day for an estimated population of about 400 000 by the year 2010
- Demand level liter/capita/day of 2000 and 2010
- Lor State Water Board’s operational and commercial statistics- Project Completion report
Alleviate the shortage of drinking water required for domestic, social and economic uses by improving the water treatment facility and the reticulation systems
- Capacity of the water treatment facility to increase by 45 500 m3/day.
- Lor State Water Board’s operational and commercial statistics- Supervision reports.
2nd Result- Activities
The Water Project Toolkit suggests checklists in order to make sure that the Evaluation Report contains the necessary information. These can be used to help build your evaluation tool by suggesting what should be considered in the evaluation of a project. Based on the above mentioned case study, the following table shows how to use the Water Project Toolkit checklist and what should be done to improve the impact of the project.
Evaluation main findings
Water Project Toolkit checklist (chapter 10)
The project goal of improving the health of people was relevant to the country’s overall macro-economic policy but the achievement was only partial as the benefits did not fully reach the poor.
Sector issues and institutional arrangements were not fully dealt with at the time of project formulation.
Project objectives were in line with water sector goal but lack integrated approach.
Provision should be made for effective inter-agency and inter-sector planning. If this was weak, lessons should be learnt for the future. Therefore examine:
p.193 Institutional and management Principles
Implementation of the plant construction process was satisfactory; however some original components were not implemented, such as the replacement of 50 km of distribution lines and the provision of technical assistance for institutional development.
Poverty alleviation and gender:
Women have benefited from the project, and the attendance to school of children, particularly girls, improved considerably in areas served by the project (Ministry of Health statistics). However the service’s coverage of the urban poor is inadequate. Provision of consumer connections in high density areas lagged behind those provided to the better-off sections of the community.
! No baseline data or indicators were set at the programming and identification stage to really evaluate the poverty alleviation and gender impact!
The additional water supply generated additional wastewater which was not addressed. Lack of proper sanitation and drainage affected some areas and helped spread disease.
Transfer of technology:
Little has been achieved; qualified staff are still made up of only a few top-level officials. Disbursements by components were not strictly observed and funds allocated for technical assistance were used to cover cost overrun in the physical component of the project.
Ownership by stakeholders and user groups of services provided by the project is essential in ensuring sustainability. Therefore examine:
Evaluation must determine to what extent intended social development has been achieved and what unexpected impacts may have occurred. Therefore examine:
Environmental damage may result because insufficient time and money is invested in collection and analysis of data during and after. Therefore examine:
In evaluating the appropriateness of technology and its influence on the wider results of the project, the accuracy of underlying data is critical. Therefore examine:
p.194 Social Principles
p.195 Environmental Principles
p.196 Technological Principles
Inadequate baseline data to calculate the Economic Internal Rate of Return (EIRR)
The expected indirect benefits of the project of spurring the establishment of industries by removing the major constraint of a lack of reticulation system did not materialise.
! No baseline data or indicators were set at the programming and identification stage to really evaluate the socio-economic impact of the project!
Sub-optimal revenue collection revenues cannot cover the O&M costs. Most of the consumer water meters that were procured as part of the project had not been installed and therefore it was impossible to recover charges for water supplied.
Projects must demonstrate economic benefit and financial accountability if they are to be sustained over the long term. Therefore examine:
p.194 Economic and Financial Principles
Operation of the pumping plant suffered from lack of spare parts arising from a lack of proper assessment of requirements and lead times.
- Premature rusting of pipes occurred because of a change in specifications of transmission and distribution without adequate safeguards, sometimes resulting in the contamination of treated water and water leakages.
Although progress has been made to increase water tariffs, the current rates did not cover the long-run marginal cost of supply. Commercial and accounting capabilities of the Lor State Water Board remained extremely weak, its finances were fully stretched and its cash flow was restricted.
In evavuating the appropriateness of technology and its influence on the wider results of the project, the accuracy of underlying data is critical. Therefore examine:
Technological solutions must be acceptable to the target users and compatible with the environment. Therefore examine:
Sustainability of infrastructure and equipment can only be achieved if the technical and financial requirements for maintenance are met. Therefore examine:
p.196 Technological Principles
Lesson learnt and recommendations
It is important to identify the project’s problems and pitfall but also its successes. Analyse them and identify learning points to avoid future problems and repeat the successes.
EC, 2011, Water Project Toolkit: Water resources management for sustainable development. Available online: http://www.aquaknow.net/en/water-project-toolkit
EC, 2012, Specificity of evaluation compared to monitoring and auditing. Available online: http://ec.europa.eu/europeaid/evaluation/methodology/methods/mth_vad_en.htm
UNDP, 2009, Handbook on planning, monitoring and evaluating for development results. Available online: http://web.undp.org/evaluation/handbook/documents/english/pme-handbook.pdf
UN, nd, A Manual on Monitoring and Evaluation for Alternative Development Projects. Available online: http://www.unodc.org/documents/alternative-development/Manual_MonitoringEval.pdf
Zarinpoush F., 2006, Project evaluation guide for non-profit organizations: Fundamental Methods and Steps For Conducting Project Evaluation.On behalf of Imagine Canada. Available online: http://library.imaginecanada.ca/files/nonprofitscan/en/csc/projectguide_final.pdf
 The country and state’s name were entirely invented for the purpose of the case study.