Perceived effectiveness indicators are specified as a basis for the evaluation instrument

Organisational receptivity refers to end-user’s perception of the DSTs effectiveness in terms of their day-to-day work. The inclusion of organisational receptivity as a criterion of perceived effectiveness is based on the belief that high receptivity scores from a range of organisational perspectives, i.e. practitioners, academics and policy makers, would indicate the decision support tool’s effectiveness in a collaborative planning context. Adams et al (1990) discuss how, when DSTs are viewed as providing decision support within an organisational context, the decision maker becomes a consumer of this support, and his or her attitudes and perceptions become important selection and evaluation measures. Receptivity has been defined by Jeffrey and Seaton (2004) as the extent to which there exists not only a willingness (or disposition) but also an ability (or capability) in different constituencies (individuals, communities, organisations, agencies etc.) to absorb, accept and utilise technology options. Three statements (statements 5, 6 and 7) exploring organisational receptivity were included in the evaluation instrument presented in Appendix T.

Reliance on decisions was included as a criterion of effectiveness to explore how Bns uncertainty and indeterminacy in the WDM problem domain, cited as a constraint to implementation of WDM during the knowledge elicitation presented in Chapter 3.

The evaluation instrument used three statements adapted from an evaluation instrument developed by Sanders and Courtney (1985), and Welsh (1980). The three statements (statements 8, 9 & 10) concerning reliance on decisions were included in the evaluation instrument presented in Appendix T.

Technical suitability examines the fit between the technical sophistication of a computer-based system (its capabilities) and user’s needs, and the impact of such a fit on system effectiveness (Srinivasan, 1985). Adelman (1992), whose method of evaluation focuses on the suitability of system characteristics to the specific problem, e.g. the adequacy of the selected analytical methods, software development costs, software tests and verification, and adequacy of the knowledge base exerts that ‘an

analytical method’s epistemological basis addresses the assumption made about the data, and perhaps most critically, the rule used to combine data to reach a conclusion.’ Adelman continues, giving a number of examples, ‘… decision-analytic and artificial intelligence methods typically use subjective data (i.e. judgements) whereas simulation and optimisation methods typically use objective, empirical data.

Decision-analytic methods use axiomatically based calculation, such as expected value, to combine utility judgements, which themselves are presumed (and elicited) to be consistent with the axioms of rational choice. In contrast, artificial intelligence methods such as those to be found in most expert systems, use heuristics (e.g., if-then rules) to represent how experts supposedly combine subjective data to reach a conclusion. And most simulations and optimisation methods use mathematical formulas to represent the relationships between data and perform calculations necessary to reach a solution on the basis of verifiable proofs’ (Adelman, 1992). The models used during the workshop relied on both objective data, i.e. hydrological and social survey data, and subjective data, i.e. expert judgements and data from literature, to populate the models.

Four statements relating to technical suitability (statements 11, 12, 14 & 15) were included in the evaluation instrument in Appendix T.

Strategic planning refers to how the tool integrates different water resource management issues. Although the evaluation activities and workshop modelling tasks did not specifically focus on integration, it was hoped that applying the water balance model and use of the household demand and LAC sub-models would provide opportunities to make analogies of how the tool might be applied to integrate these activities. One statement (statement 13) in the evaluation instrument explored user’s perceived effectiveness of the tool in strategic planning.

Transparency refers to the recognition that at any point in time the end-user should have access to the background information needed to understand the models they are working with, the processes represented, and the numbers generated. Without this information, models remain black boxes and learning is excluded (FutureTech, 2002). Ubbels and Verhallen, (2000) evaluated the suitability of tools for specific user groups and decision making phases for collaborative planning processes using characteristics including user friendliness, transparency, flexibility, and the way the effects of possible actions are estimated.

Two statements exploring end-user’s perceptions of transparency of the DST (statements 16 & 17) one adapted from an evaluation instrument developed by Sprague and Carlson (1982), and a further one adapted from Jenkins and Ricketts (1979) were included in the evaluation instrument presented in Appendix T.

Learning refers the effectiveness of the support tool in teaching users about the problem domain. Welsch (1980) and Sanders and Courtney (1985) included learning as a dimension to explore how their tool supported dialogue and enquiry with other decision-makers. Watkins and Marsick’s Dimensions of the Learning Organization Questionnaire (DLOQ) (Watkins and Marsick 1997; 2003) provided a second source of material to design questions to elicit user’s perceptions of the tools effectiveness in providing learning support. Two statements relating to learning (statements 18 & 19) were included in the instrument presented in Appendix T.

Ease of use refers to the ability of the support tool to present information to a decision maker in ways that are clear and familiar, and that permit rapid comprehension and has been included by a number of researchers in evaluating DSTs (Sprague and Carlson, 1982). Ease of use is also included in the evaluation instrument as a checking mechanism to indicate if responses to statements regarding the other six criteria of perceived effectiveness were influenced user’s experiencing difficulty in applying the tool. Research reported by Sanders and Courtney (1985) showed a negative correlation between difficulty in using DSTs and overall satisfaction with the tool. Srinivasan (1985) also reported that lower perceived effectiveness correlated with time spent using the DST in their study. Both results imply that a correlation may exist for some users between satisfaction and the difficulty in applying the tool for a specific task. One statement was included in the evaluation instrument (statement 20, Appendix T) to elicit user’s perceptions of ease of use.

In addition to statements relevant to the perceived effectiveness indicators described above, a number of questions were included in the evaluation survey regarding the informed practitioners perceptions of the existing decision process compared with their experience of using the Bayesian network models during the workshop.

Kottemann and Davis (1991) use the term decisional conflict to refer to the negative affective state experienced by a decision maker as a result of making explicit trade-off judgments among alternatives. There are several studies that give evidence of the

decision conflict originating from analytical methods used in decision making processes (e.g., Bettman et al., 1993; Luce et al., 1999; Scholten, 2002). Janis and Mann (1977) theorize that trade-off conflict is a major source of decisional stress.

Aloysius et al (2006) measured decisional conflict among users of different types of analytical techniques used in DSTs. They found that some analytic methods used in DSTs, e.g., pair-wise comparisons, require users to make trade-offs leading to greater decisional stress due to the decision conflict, whereas other analytic methods, e.g., those giving output as absolute measurements, result in less decisional conflict.

A large body of evidence exists (Aloysius et al, 2006; Shugan, 1980; Bettman et al., 1990; Chatterjee and Heath, 1996) linking higher conflict tasks with more cognitive effort for the decision maker, as they attempt to better confront the trade-offs inherent in a multi-attribute problem. As a result users who perceive high levels of decision conflict will also perceive the task to be more effortful.

When decision making tasks are perceived to be higher in effort, decision makers tend to perceive that the results of their decision making are lower in accuracy, due to the increase in perceived decision difficulty (Peterson and Pitz, 1988; Chatterjee and Heath, 1996). It has also been suggested that the higher perceived effort may reflect some limitations in their own ability in the task domain (Reeder et al., 2001).

Following on from this body of research the questionnaire first explores individual’s perception of decision conflict, effort and confidence in the existing decision process in Sofia. In the final section of the questionnaire, each end-user is asked about their perceptions of decision conflict, effort and confidence when using the Bayesian network models during the workshop tasks. The results are compared for significance of variance to provide evidence for a discussion about how Bns facilitate these aspects of the decision process.

7.1.2 Eliciting perceived effectiveness scores using the evaluation