19 Junho 2019

Using Indicators to Measure Progress and Performance

Author/Compiled by
Beat Stauffer (seecon international gmbh)

Executive Summary

Progress and performance must be measured to attest a development in a (long term) project. With few, but carefully selected indicators, it is possible to get a good overview on the progress and performance. Indicators can be used in many different areas including water and sanitation. Here, you will find how indicators work and how they are developed.

Advantages
Indicators can be adapted to the circumstances
It is not necessary to measure every single parameter to get a “bigger picture” over the whole progress
Failures can be detected and eliminated
Disadvantages
Too many indicators; A set with a large number of indicators will tend to clutter the overview
Measuring indicators is a long-term exercise. If the indicators are not defined properly, and if measuring is not done correctly and conscientiously, it will not work

Introduction

Factsheet Block Body

“How do we know process is being made? How can we know if an intervention is making our local water and sanitation system more sustainable?” These are very important questions regarding projects or programmes in water and sanitation. Therefore a suitable method is needed to make the development visible and measure the progress.

Indicators are one approach to measure progress.The term "minimum indicators" is used in recognition that:

  • It is better to start with a small set of indicators that are feasible to monitor and to improve over time.
  • There are many other indicators that could be used to measure progress beyond this basic level.

 

Indicators provide an effective tool to measure progress and performance. An indicator is the representation of a trend tracking the measurable change in a system over time. Generally an indicator focuses on a small, manageable set of information that gives a sense of the bigger picture. Therefore it can be seen that there is no need to measure everything. Furthermore the choice of indicators is important as to whether it gives sufficient ‘sense of the bigger picture’.

Only well selected indicators can assist the process to maintain a focus on the important work areas and take strategic decisions to address problem areas. That makes a project sustainable and allows responsible persons to act.

Use of Indicators

Factsheet Block Body

Indicators are useful to:

  • Measure progress over time against various water and sanitation objectives providing information relevant to policy.
  • Measure performance against a target to evaluate the effect of policy actions and plans.
  • Present information to the public or stakeholders in a simplified way.
  • Identify areas for increased attention by an organisation.

 

Indicators Have Two Core Functions:

Factsheet Block Body
  • To provide system information to inform the project team (responsible organisation), the public and policy makers.
  • To translate data into policy relevant information. That is, they describe, show trends and communicate the results of implementing objectives.

 

Criteria for Developing Indicators

Factsheet Block Body

The most important point in developing indicators is not to be too ambitious. Start with what can be realistically done or else failure is guaranteed.

Millennium development goal 7, target 10 and its indicators. Source: UN MILLENNIUM PROJECT (2006)
Millennium development goal 7, target 10 and its indicators. Source: UN MILLENNIUM PROJECT (2006)

 

a) Simple, Easily Measured, Understood and Applied

The data used for indicators should be in a format that is easy to use. The more complex the indicator the less useful it will be.

 

b) As Few as Necessary

The capacity to measure and report is usually limited by financial and human resources.Being burdened with an excessive number of indicators may mean that the system fails to achieve the expected benefits or does not work at all. A set with a large number of indicators will tend to clutter the overview it is meant to provide.

 

c) Use Existing Information Where Possible

It is preferable that the information needed to measure an indicator is available through existing data sources and monitoring programmes or that data collection can occur through existing programmes. This will improve the cost effectiveness of the system.

 

d) Relate at the Appropriate Scale

An indicator should be related to the specific situation it is "indicating" information about. The indicator should be measurable at an appropriate scale.

 

e) Detect Change

The indicator should be able to detect change. If it does not reflect change because it was poorly selected or the situation has changed then another indicator should be identified.

 

f) Comparable, Repeatable and Defensible Between Sites and Times

It is an advantage to use indicators which are comparable between similar projects in different countries. That allows transboundary comparisons and assures the choice of the indicator.

Note: Indicators are useful tools for measuring progress and motivating action in specific areas. They should be limited to those that can reasonably be measured. Start small, build up gradually.

Typical Indicators to Measure Sustainability in Sanitation and Water Management

Factsheet Block Body

(Adapted from MUGA and MIHELCIC 2008)

  • Functional indicators (e.g. adaptability, durability, and reliability of the system)
  • Economic indicators (e.g. capital costs, operation and maintenance costs and user costs)
  • Environmental indicators covering resource use (e.g. nutrient reuse) and emissions (e.g. emission of carbon dioxide)
  • Social-cultural indicators (e.g. public participation, acceptance, and stimulation of sustainable behaviour)

 

Typical indicators in the field of water and sanitation. Source: UN-WATER / AFRICA (2006) 
Typical indicators in the field of water and sanitation. Source: UN-WATER / AFRICA (2006)

 

Depending on the scope of an intervention in the field of sustainable sanitation and water management, different indicators will be applied. Some typical specific examples to measure sustainability are:

 

Water Sources Management

  • Water withdrawal: absolute or per-person value of yearly water withdrawal gives a measure of the importance of water in the country's economy.
  • Ground water level development (rise or fall of ground water levels, pollution).
  • Water sources quality (e.g. turbidity, minerals content, pathogens, chemical quality).

 

Water Purification

  • Treatment efficiency indicator (e.g. removal of indicator bacteria).

 

Water Use

  • Percentage or number of people not served with improved drinking water and extension of piped water supply.
  • Domestic water use efficiency (per capita use of water).
  • Agricultural water use efficiency and efficacy (water demand per irrigated area).
  • Industrial water use.
  • Awareness and actual use of water-saving appliances.

 

Wastewater Collection

  • Percentage of wastewater collected to total wastewater production.

 

Wastewater Treatment

(Adapted from MUGA and MIHELCIC 2008)

  • Percentage or number of people not served with basic sanitation.
  • Number of people affected by water borne diseases (e.g. diarrhoea, cholera).
  • Investment in drinking water supply and sanitation.
  • Removal of pollutants.
  • Awareness and actual use of safe sanitation systems.

 

Reuse and Recharge of Nutrients and Water

  • Percentage of reclaimed water used in agriculture to total agricultural water use.
  • Percentage of nutrients used from recycled sources.

 

Of course, this list is by no means exhaustive. Furthermore, also other, less technical indicators (such as e.g. hygiene behaviour, knowledge on water related diseases etc. could be measured).

Applicability

Indicators are useful to measure progresses and performances of (long term) projects in water and sanitation. It makes successes or failures visible and with a precise influence on the indicators problems can be solved.

Library References

Integrated Water Resources Management for River Basin Organisations – Training Manual

This training manual about Integrated Water Resources Management contains a module on indicators and its application.

TAYLOR, P. LIDEN, R. NDIRANGU, W. JIN, L. (2008): Integrated Water Resources Management for River Basin Organisations – Training Manual. Pretoria: International Network for Capacity Building in Integrated Water Resources Management (Cap-Net) URL [Accessed: 19.06.2019]

Distribution System Indicators of Drinking Water Quality

This paper compiles available information on indicators of drinking water quality within potable water distribution systems. The indicators include microbial and non-microbial parameters, for which sample collection and analyses could be performed to identify existing or potential problems, as well as other methods or tools that may similarly function as problem indicators.

EPA (2006): Distribution System Indicators of Drinking Water Quality. URL [Accessed: 19.06.2019]

Sustainability of Wastewater Treatment Technologies

This paper presents a set of indicators that incorporate environmental, societal, and economic sustainability that were developed and used to investigate the sustainability of different wastewater treatment technologies, for plant capacities of 5 million gallons per day.

MUGA, H.E. ; MIHELCIC, J.R. (2008): Sustainability of Wastewater Treatment Technologies. In: Journal of Environmental Management : Volume 88 , 437 - 447. URL [Accessed: 22.04.2012]

African Water Development Report 2006 – Chapter “Indicators - Measuring the Progress of the African Water Vision

The Africa Water Vision 2025 was formulated to guide the development and management of water resources for sustainable development. A number of indicators were selected along the lines of the methodologies developed for the World Water Development Report in order to quantitatively measure progress in the implementation of water-related projects.

UN-WATER AFRICA (2006): African Water Development Report 2006 – Chapter “Indicators - Measuring the Progress of the African Water Vision. Addis Ababa: UN Water/Africa URL [Accessed: 19.06.2019]
Further Readings

Distribution System Indicators of Drinking Water Quality

This paper compiles available information on indicators of drinking water quality within potable water distribution systems. The indicators include microbial and non-microbial parameters, for which sample collection and analyses could be performed to identify existing or potential problems, as well as other methods or tools that may similarly function as problem indicators.

EPA (2006): Distribution System Indicators of Drinking Water Quality. URL [Accessed: 19.06.2019]

Sustainability of Wastewater Treatment Technologies

This paper presents a set of indicators that incorporate environmental, societal, and economic sustainability that were developed and used to investigate the sustainability of different wastewater treatment technologies, for plant capacities of 5 million gallons per day.

MUGA, H.E. ; MIHELCIC, J.R. (2008): Sustainability of Wastewater Treatment Technologies. In: Journal of Environmental Management : Volume 88 , 437 - 447. URL [Accessed: 22.04.2012]

Integrated Water Resources Management for River Basin Organisations – Training Manual

This training manual about Integrated Water Resources Management contains a module on indicators and its application.

TAYLOR, P. LIDEN, R. NDIRANGU, W. JIN, L. (2008): Integrated Water Resources Management for River Basin Organisations – Training Manual. Pretoria: International Network for Capacity Building in Integrated Water Resources Management (Cap-Net) URL [Accessed: 19.06.2019]

Review of Frameworks for Technology Assessment

A WASHTech literature review of existing frameworks for technology assessment reveals that there is a gap for a WASH technology assessment tool and a WASH technology uptake tool. The authors of the review, which supports the development of WASHTech’s Technology Assessment Framework, (TAF), conclude that a computer tool based on an algorithm is not appropriate because it is too rigid. Choosing a manageable number of appropriate indicators is key for assessing new technologies.

OLSCHEWSKI, A. DANERT, K. FUREY, S. KLINGEL, F. (2011): Review of Frameworks for Technology Assessment. WASHTech Deliverable 3.1. St. Gallenand The Hague : Swiss Centre for Development Cooperation in Technology and Management (SKAT) and International Water and Sanitation Centre (IRC) URL [Accessed: 26.05.2019]

Exploring Changes in Open Defecation Prevalence in Sub-Saharan Africa Based on National Level Indices

This study estimates the changes in open defecation prevalence between 2005 and 2010 across countries in Sub-Saharan Africa. It also explores the association between national level indices and changes in open defecation prevalence and assesses how many countries can achieve “open defecation free status” by 2015.

GALAN, D.I. ; KIM, S. ; GRAHAM, J.P. (2013): Exploring Changes in Open Defecation Prevalence in Sub-Saharan Africa Based on National Level Indices. In: BMC Public Health: Volume 13 , 1-12. URL [Accessed: 28.08.2013]

Outcome and Impact Assessment in International Development

These guidelines are designed to help project managers to assess the outcomes of their projects and programmes. They demonstrate how development agencies can implement an appropriate outcome and impact assessment system.

ZEWO (n.y): Outcome and Impact Assessment in International Development. Zewo Guidelines for Projects and Programmes. Zurich: Schweizerische Zertifizierungsstelle fuer gemeinnuetzige, Spenden sammelnde Organisationen (ZEWO) URL [Accessed: 04.10.2013]

Achieving Sustainability: Changing the Ways in Which we Define Success

Implementing agencies like WSUP, Water For People and IRC are of course accountable to their funders, including major bilaterals and foundations. And naturally, these funders must track the effectiveness of their spending. But short budget cycles and the need to demonstrate “value for money” can often encourage over-simplistic measurement of success in terms of short-term outputs, rather than genuinely sustainable services. This note proposes some ways forward.

COWLING, R. NORMAN, G. (2013): Achieving Sustainability: Changing the Ways in Which we Define Success. (= Perspective , 1 ). London: Water and Sanitation for the Urban Poor (WSUP) URL [Accessed: 25.11.2013]
Case Studies

African Water Development Report 2006 – Chapter “Indicators - Measuring the Progress of the African Water Vision

The Africa Water Vision 2025 was formulated to guide the development and management of water resources for sustainable development. A number of indicators were selected along the lines of the methodologies developed for the World Water Development Report in order to quantitatively measure progress in the implementation of water-related projects.

UN-WATER AFRICA (2006): African Water Development Report 2006 – Chapter “Indicators - Measuring the Progress of the African Water Vision. Addis Ababa: UN Water/Africa URL [Accessed: 19.06.2019]

Measuring Sanitation: Outcomes as well as outputs

At the second SACOSAN conference held in Islamabad in 2006, countries in the region committed to work together to identify a new set of indicators for sanitation and use these to report on progress at the next SACOSAN to be held in India in 2008. This paper has been prepared as a thinkpiece for consideration by governments and stakeholders working on possible indicators.

WICKEN, J. (2008): Measuring Sanitation: Outcomes as well as outputs. London: WaterAid URL [Accessed: 15.11.2010]
Training Material

Alternative Versions to