management implementation
Introduction
In Chapter 8 the findings of the case study fieldwork are discussed in the context of how Bayesian network modelling addresses issues of validity and legitimacy. A philosophical debate on how to conduct Information Systems (IS) research, i.e.
positivism vs. interpretivism, has been the focus of much recent attention (Robey, 1996; Klein and March, 1995; Weber, 2003). Hevner et al. (2004, p75) write that “…
the major emphasis of this debate in IS research lies in the epistemologies of research; that is, somewhere some truth exists and somehow that truth can be extracted, explicated and codified.” The behavioural-science paradigm seeks to find
‘what is true’. In contrast, the design-science paradigm seeks to create ‘what is effective’.
Determining what is ‘true’ and, at the same time, what is perceived to be ‘effective’ in terms of support tool performance are central issues that overlap with historical debates (e.g. Rosenhead, 1989; Boulaire, 1992; Landry et al., 1996) about the importance of validity and legitimacy in IS research. The debate has generally pitched one against the other in a hierarchy, e.g. knowing ‘what is effective’ is dependent on knowing ‘what is true’, whilst recognising that what is perceived to be immediately ‘true’ can sometimes be affected by individual’s judgements about what is ‘effective’. For water demand management, where there is an emphasis on forecasting to design the future management of river basins, a further dimension arises because the aim of any research in this area is judging what will be effective in the future which is dependent on knowing what will be true. It might be suggested that this is why there is an emphasis on the need to manage risk and uncertainty in the WDM problem domain.
The discussion in Section 8.1 uses the results of the case study fieldwork as evidence of how Bayesian modelling addresses the parallel need for validation and
legitimisation of models. Section 8.2 refers to the evaluation results as a basis for a discussion about the potential pay-off of using Bns from different stakeholder perspectives. In Section 8.3 lessons are drawn from the case study fieldwork to support a discussion about legitimisation in the context of a country in the midst of economic and social transition.
It should be pointed out that the discussions below are not an attempt to gauge the validity or legitimacy of the Bn models developed during the case study fieldwork but rather to draw lessons from the research about how attributes of Bayesian modelling address validation and legitimisation issues, thereby supporting integration of science and practice.
8.1 Bayesian modelling to facilitate validation and legitimisation of the water demand management decisions
8.1.1 Validation
Model validation is an essential step in the modelling process to build-up confidence in the current model or to allow selection of alternative models or model parameters (Tedeschi, 2005). Because the WDM problem domain is characterised by complexity and non-repeatability of events problems arise when attempting to evaluate the validity of model outputs in terms of their accuracy and precision. As discussed below, some of the problems that arise are a result of factors associated with WDM implementation, whereas others are due to a combination of the problem domain and the modelling / analytical method used.
8.1.1.1 Accuracy and precision
Accuracy measures how closely model-predicted values are to the true values, whereas precision measures how closely individual model-predicted values are to each other. In other words, accuracy is the model’s ability to predict the right values and precision is the ability of the model to predict similar values consistently. Figure 8.1 (below) from Tedeschi (2005, p5) illustrates the difference between accuracy and precision using the analogy of target practice.
Figure 8.1. Schematic of accuracy versus precision: Case 1 is inaccurate and imprecise, case 2 is inaccurate and precise, case 3 is accurate and imprecise, and case 4 is accurate and precise
Testing a model usually involves comparison of predicted outputs with a real world
‘control’ sample. For implementation WDM strategies non-repeatability of events limits how models can be tested both at the legislation and design stages. For example, during the design stage testing the accuracy and precision of the household water demand or water saving forecasting model presented in Chapter 6, Section 6.4, is challenging but could be achieved using a control sample, as has been demonstrated in former studies described in Chapter 1, Section 1.2.2.1 (e.g.
Turner et al., 2005). However, testing the uptake forecasting model presented in Chapter 6, Section 6.3, is more problematic because implementation conditions, i.e.
household/ demand variables profiles for a population will never be repeated.
For the legislation stage, problems of forecasting water availability (i.e. reservoir level forecasts) can be partially addressed by using historical data and hydrological modelling, as discussed in Chapters 4 & 5, although the rare or one-off nature of droughts means that such forecasting models are still difficult to validate. Developing
Source: Tedeschi, 2005
models for forecasting bulk water savings and the costs and benefits arising from these savings faces major problems of non-repeatability because implementation of WDM strategies in a city is a one-off event. However, the premise of the ‘design’
models in Chapter 6 is that disaggregating uptake and water savings to the neighbourhood scale will provide information to support the detailed implementation of measures and the design of relevant uptake mechanisms.
Model testing is commonly used to prove the rightness of a model and the tests are typically presented as evidence to promote their acceptance and usability. However as a number of authors have commented (Sterman, 2002; Tedeschi, 2005; van den Hove, 2007) the understanding and acceptance of the wrongness and weaknesses of a model strengthens the modelling process, making it more resilient and powerful in all aspects during the development, evaluation, and revision phases. Rather than ignoring the fact that a model may fail, design evaluations to identify and incorporate the failures of a model strengthen the learning process. Sterman (2002) points out that in systems thinking, the understanding that models are wrong and acceptance of the limitations of our knowledge is essential in creating an environment in which we can learn about the complexity of systems. The findings of the technical evaluation that Bns offer support for identifying research priorities and evaluating confidence in data, and the findings of the end-user evaluation regarding their transparency for analysing strengths and weights of causal relationships both demonstrate their potential as an interface for communicating research issues such as uncertainty and data availability to a wider audience.
In the following section the importance of data and information processing for validation in complex problem domains such as water demand management is discussed. The suitability of Bns for supporting validation tasks is reviewed citing examples of how Bayesian modelling was applied during the case study fieldwork.
8.1.1.2 Complexity and uncertainty
Uncertainty is considered to be a property of the environment resulting from two powerful forces: complexity and the rate of change. Complexity refers to the number and diversity of the elements in an environment and the rate of change refers to how rapidly these elements and the interactions between them change (Sahota, 2004).
Duncan, (1972) showed that what affects organisations is not the environment so much as the decision maker’s perceptions of how uncertain the environment is; these concepts are summarised in Figure 8.2, below.