Monitoring and Evaluation in the WASH sector

Over the last 50 years, there appears to be a continuous and repetitive debate around ‘what’ should be monitored in the WASH sector. Sometimes the discourse identifies the pros and cons of one indicator or index versus another as Fietelson & Chenoweth (2002) report. Alternatively debate centres on rationalising the need to create new indices to address the gaps in what is not being measured (Caplan & Jones, 2002). In other cases acknowledging that different requirements of M&E at different levels in turn influences ‘what’ is being monitored or evaluated (Cotton & Bartram, 2008) is also reported.

Whether the interest is related to the setting of the post 2015 MDG targets, defining national sector policy and strategic plans, or simply wanting to know when there will be a continuous, safe supply of affordable water, monitoring and evaluation is integral. Pragmatically, each of these three interest areas should be linked via aggregated data sets generated through the activities of monitoring and evaluation: the process of determining outputs and outcomes, ideally relative to inputs.

According to Fietelson & Chenoweth (2002), also recognised by Nelson and Zadek (2000) and Palme &Tillman (2009), “indicators are instruments that are used for communicating key information in a simplified form to policy makers and the general public”. They go on to suggest that indicators serve the function of simplification and can be used to reduce the confusion potentially caused by large amounts of environmental and economic data’ as well as “galvanize decision makers into action”. Gallopin (1997) goes on to recognise that “useful indicators must be measurable and….the data must be either obtainable through measuring and monitoring, or be readily available” (Fietelson & Chenoweth, 2002).

Various articles, guidelines and handbooks, on the new and expanding number of indicators, have continued to be published for many years (Hicks & Streeten, 1979; WHO, 1985b; Mosse & Sontheimer, 1996; EC, 2006; AfDB, 2011), with

Narrative also continues in terms of acknowledging differences in definitions and interpretations of indicators (Box 2-3).

Box 2-3 References regarding definitions and complexity of indicators

Furthermore, discourse extends to the value of composite indicators – a sort of aggregating of data - as reported by Saisana & Tarantola (2002). However as reported by Dahl (1997), these are sometimes identified as being “fraught with conceptual and practical problems”. Despite the challenges of composite or a multi-dimensional index combined with the increasing number of indicators being made available, no-one has yet come up with what could be seen as a proverbial ‘silver bullet’ index.

As Fietelson & Chenoweth (2002) recognise, when discussing the Water Poverty Index as a new index, the WPI has a limited purpose – to identify the degree to which countries or regions are likely to face problems in addressing their water supply needs, taking into water quality and affordability issues. It will not distinguish between cases where failure to provide service delivery is the result of mismanagement, power structures, lack of technical and administrative capacity amongst other factors. This acknowledgement seems to suggest that monitoring improved and sustained service delivery will need more than one index. This is further supported by the claim of the suggested benefits of having qualitative data in addition to the more frequently reported (traditional) quantitative data (Oakley, 1988; Sijbesma & Postma, 2008).

WHO (1984): “…the concept of safe and adequate drinking-water supply and

excreta disposal may have different interpretations in different regions and countries. In setting targets, countries may wish to assign more accurate boundaries to safety and adequacy…..”

Aiga (The Lancet) (2003): “Of 45 countries jointly reviewed by WHO and UNICEF

in 1996, 42 used distance-based definitions, 38 used water quantity-based

definitions, and two used time-based definitions, implying that 37 countries defined access with a combination of these three factors”. (Referring to the standardisation of the definition of access to safe water)

Dar & Khan (2011): “It is undoubtedly difficult to summarise a complex target such

New indicators or indices, often created or developed to fill a gap – perceived or otherwise – could also be considered as adding to the burden of data collection. Whilst recognising the positive intentions of the creator (Box 2-4), the questions, “who is the creator?” and “to what extent is the creator the user?” coupled with “where is the demand coming from?” should be considered and better understood.

Box 2-4 Suggestions for new indicators and guidelines

As suggested by Pybus & Schoeman (2001), part of the problem of the indicators being used, in this case within a performance management framework, is that ‘we’ are looking at what are the data points needed to report against the indicators as set by national and global level, rather than considering what the data points are that are needed at the local level for informed decision making to ensure sustainable service delivery. The concern of where the responsibilities lies in terms of determining what should be monitored is echoed by Lucas et al (2004), Wambulwa (2008) and Edmunds & Marchant (2008).

Acknowledging the claim that different levels need different information the question of how data points can be aggregated up and combined with other sector data, recognising the complex and integrated nature of WASH sector

Hicks & Streeten (1979): “This new focus on meeting basic human needs requires

an indicator or set of indicators, therefore, by which deprivation can be judged and measured, and policies directed at its alleviation and eradication can be initiated and monitored”. (Moving from GDP)

Mosse & Sontheimer (1996): “Without appropriate feedback, none of the parties

concerned with project outcomes could make appropriate, informed decision about whether and how to adjust project design or implementation arrangements to better achieve a project’s intended objectives” (Referring to the introduction of Performance monitoring indicators)

AfDB (2011): “The Handbook is intended to serve as a main reference document to

guide African countries and regional institutions in collecting standardized and comparable data on infrastructure”.

service delivery9, to simplify the messages for management and policy decision making, seemingly remains.

Alternatively is the perception of the burden of data collection – one that is a global phenomenon – compounded through an avoidable set of factors such as improved data confidence, improved knowledge and understanding of the purpose of monitoring and evaluation. Furthermore, could it be allayed by improved partnerships (as argued by Caplan & Jones, 2002) and improving coordination of activities in addition to gaining some perspective in terms of thinking the ‘new’ will overcome the challenges, without having sufficient evidence that it will.

As the research of Palme & Tillman (2009) proposes “there is a mismatch between what the information senders want to convey to the receivers and what the latter actually want in terms of information”. In other words, those who are collecting the data and those receiving the data want different things. Who should lead or champion the dialogue or should there be a dovetailing from both sides, through an iterative process? Moreover, it could be that more effort is needed in understanding what the driver and incentives are for data collection. Alternatively, perhaps we should stop definitively or pause and reflect back to first principles to remind ourselves what an indicator should be – in order to simplify.

Another dimension to consider, in terms of the plethora of information available, is that raised by Berg & Marques (2011) – relevance and importance of the topic. They claim that academic research relating to water and sanitation utility benchmarking for example – performance monitoring approach - is merely ‘dipped into’ and only a few researchers engage in sustained research. As part of their literature review the authors also observe that very few articles relate to quality of service, financial sustainability and determinants of economic sustainability, considered today as a fundamental purpose of benchmarking.

9

Complex in terms of often mandated as the responsibility of multi-sectorial government ministries coupled with, in lower-income countries especially, multiple funding streams via donor and

The issues are linked to data availability levels and the need for closer links between empirical researchers, policy analysts and theorists as also referred to by von Neumann, (1947).

These latter two aspects alone raise questions over the purpose of research, roles and responsibilities and whether it is demand driven as well as the extent of reporting and use of data, once collected. A similar set of questions are evident within the context of monitoring and evaluation in the WASH sector. A body of literature seemingly exists on framing the purposes (‘why’) behind, or reasons for monitoring and evaluation (Casley & Kumar, 1988, IPDET 2007, Weiss 2004) and yet conversely discourse asking questions about efficacy of use of M&E data is somewhat opaque.

Equally perverse is despite a corpus of literature on preferred methods and approaches (‘how’) such as that reported by Garfi & Ferrer-Marti (2011) and Nardo et al (2005), including claims that new and emerging technological solutions are improving and easing the burden of data acquisition discourse simply asking, let alone answering, questions about how much M&E is costing and the cost-benefit analysis of the tools, methods and approaches used, is scarce.

Within the UK as part of a recent study into future priority research (Brown et al, 2010), in response to the increasing emphasis being placed over the last 10- years on ‘evidence-based policy’, a reference was provided from the House of Commons Science and Technology Committee, dating back 2006. The reference, more broadly suggests the necessity of investment in “research to underpin policy making and embed ‘horizon scanning’ into the process to combat the short-term nature of the political cycle”.

The questions of ‘how much is M&E costing?’ and ‘what is the efficacy of use of M&E data?’ are also two seemingly fundamental questions that need to be asked and in turn answered, in a day of austerity, where ‘value for money’ is increasingly demanded.

Within the previous paragraphs some of the challenges faced in terms of monitoring and evaluation in the WASH sector have been identified, however, in terms of global monitoring and the shift from water rights to the Human Right to Water and Sanitation adds another dimension. As reported in UN (2007) explicitly and implicitly water and sanitation has been referenced in human rights treaties, principles and guidelines, declarations and resolutions. In turn obligations for monitoring the Human Rights agenda are set out, allowing specifics to be determined at a country level. However, in 2010, two significant resolutions were made by the UN Generable Assembly and the Human Rights Council.

The first10, was the formal recognition for the right to water and sanitation acknowledging “that clean drinking water and sanitation are essential to the realisation of all human rights” (UN-Water, n.d). The second11 _{“affirms that the}

rights to water and sanitation are part of existing international law and confirms that these rights are legally binding upon States” (UN-Water, n.d). Furthermore, the resolution “calls upon States to develop appropriate tools and mechanisms to achieve progressively the full realization of human rights obligations related to access to safe drinking water and sanitation, including in currently un-served and under-served areas” (UN-Water, n.d). Implicitly this includes monitoring and evaluation tools. It is with this in mind that the discussions around post-2015 and the sustainable development goals, also include the progressive realisation of the human right to water and sanitation.