Methodology

The study uses what it terms Multi-Criteria Mapping (MCM) to make a comparative appraisal of GM herbicide tolerant oilseed rape with other technical and policy options. A panel of twelve people, all knowledgeable about the issue in question but representing a wide range of perspectives, was asked to undertake a simplified MCDA application to assess the relative performance of six basic options determined by the research team (organic, integrated pest management, conventional and GM strategies under

different regulatory regimes), plus up to six others that they themselves were free to introduce.

The analysis essentially worked through the eight steps of the MCDA process set out in Figure 6.1. A key difference, however, relative to standard decision analysis, is that each individual was guided through his or her own individual analysis in a separate 2–3 hour session. Thus, from the outset, the aim was not to achieve a consensus view within the group of twelve on the relative attractiveness of the options, but to expose the variety of views and to try to understand better where the differences were most marked and why.

Step 1: Establish the decision context

The decision context was established through the prior contact to arrange the MCDA interview, pre-circulated written material, and then by direct discussion between the researchers and the person concerned. Participants were chosen based on their knowledge of the subject area and in order to reflect a wide range of perspectives. Clearly it was necessary to explain carefully the purpose of the exercise, the nature and role of the MCDA application, and to cover confidentiality and other matters highlighted by the sensitivity of this particular debate.

Step 2: Identify the options to be appraised

These consisted of the six basic options plus any others identified by the interviewee. Nine of the twelve participants added a total of seventeen options.

Step 3: Identify objectives and criteria

Participants were asked to identify up to twelve criteria. Perhaps because each MCDA application was a separate, individual one, little emphasis seems to have been placed on the explicit identification of objectives to which the criteria related. A basic check for preferential independence was undertaken.

Some grouping of criteria was done for later analysis, but not directly to support the MCDA modelling process itself.

Step 4: Scoring

Subjects had some difficulty with this part of the analysis, possibly because of the scientific complexity and high levels of uncertainty surrounding some of the criteria and the scope for context-dependent variability. As a means of exploring uncertainty and variability, participants were asked to state both a ‘high’ and a ‘low’ estimate for each criterion score.

Step 5: Weighting

Swing weighting was not employed in this particular study. Rather,

participants generally adopted a more intuitive, ad hoc approach. However, they were made aware of the interdependence between weights and

scores, so that weights were, in principle, related to the degree of difference between the best and worst performance levels considered for each criterion, as is required for the MCDA model.

Step 6: Combine the weights and scores

This was done using the conventional linear additive model described in Appendix 4.

Step 7: Examine the results

See below

Step 8: Sensitivity analysis

See below.

9.3 Findings

Because the aim of the MCM was not primarily the conventional identification of good options, the analysis and interpretation of results departed from what would normally have been done in steps 7 and 8 of a standard MCDA. The principal analyses that were undertaken for this particular application concerned: the types of criteria nominated by the participants; various sensitivity tests on option rankings; and a consideration of the desirability of developing diverse portfolios of options, rather than the ‘single best solution’ which is implicit in the basic implementation of MCDA. Regarding the first of these, it was found that most participants identified criteria that could be categorised under one of five main headings:

environment, agriculture, health, economic and social. Selections tended to reflect participants’ own professional interests. However, many of these criteria lay outside the scope of official risk assessments.

Building in part on this analysis, a second aspect of the study was a form of sensitivity testing, to explore what were the key factors determining participants’ ranking of alternatives. The study explored whether ranking changed significantly depending upon whether optimistic or pessimistic values of the scores were used and in the light of substantial changes in the order of magnitude of individuals’ weight assessments. The eliciting of ‘optimistic’ and ‘pessimistic’ scores, and the documentation of associated assumptions and conditions, played a crucial role beyond the quantitative assessment of performance. It gave a handle on the key variables in different perspectives on performance itself. Interestingly, the general finding was that there was relatively little change in ranking from either of these sensitivity checks.

However, what was very influential in deciding what options ranked highly was the choice and definition of criteria themselves. In other words, how participants ‘framed’ the problem, the overall view of the world they brought to the analysis as reflected in the mix of criteria they felt to be important, was critical. This result is quite consistent with the view of many experienced decision analysts, who would argue that time spent determining the criteria in any MCA is the most important time of all, and generally much more so than excessive fine-tuning of the numerical detail of the models themselves. The authors also discuss the implications for the practice of aggregating the criteria of different participants into an overarching value tree.

Thus one of the principal results (that different frames on decision making as reflected in attribute choice are more likely to differentiate alternatives than different weights and/or scores in a shared model framework) is reasonably well established already in the decision research literature. In this respect, it might be asked what precisely further applications of this approach to other topics might achieve. Perhaps the main benefit would be simply a demonstration effect to those responsible for decision making in the area, that stakeholders’ framing of problems differs significantly and that this in turn directly underlies the differences that are observed in preferences for alternatives.

In terms of the conduct of risk analyses and, indeed any form of multi-criteria assessment, this finding supports strongly the view that it is important to avoid premature focusing of the analysis around a single (often technical) perspective on option choice. Conventional risk assessments, because of the limited range of criteria they examine, may well be seen as flawed and unacceptable by some stakeholders. Further, the type of approach developed in MCM can also often aid the identification of robust alternatives, options that may not be the very best in any one interest group’s eyes, but which perform tolerably well across all the main perspectives that may be brought to bear on a problem. For example, here, an organic option was a robust performer, whereas the status quo option of continuing intensive agriculture (without GM) often ranked poorly.

Finally, the MCM analysis also provided a basis (described in detail in the report) to explore the attraction of selecting not a single option (often the

unnecessarily restrictive way in which many policy choices are presented) but a diverse portfolio embracing some mixture of the options under consideration.

9.4 Assessment

It is important to recognise that the way in which the MCDA pattern of thinking is used in MCM is different in important ways from the prescriptive mode of application set out elsewhere in this manual. It does not aim to identify promising options judged against criteria derived from an agreed set of objectives that reflect the values of the decision making group, or of society as a whole. Rather, its aim is to map the diversity of perspectives that may be taken on an issue, to highlight the key features underlying the differences and to provide a framework for debate. Both in terms of its identification of criteria and of the way in which the individual steps of analysis are undertaken, it does not match what would be done in a conventional MCDA application. That is not its intention.

Provided that a clear distinction is maintained such that the outcomes of an MCM are not interpreted as providing the basis for direct decision support, MCM does represent a useful extension of the basic MCDA pattern of thinking that allows it to throw light on a new set of issues. There are many other mapping procedures that allow problems to be described in formal terms and then debated (e.g., Eden and Ackermann, 1998) or decision making procedures to be analysed (Hodgkinson et al., 1999). MCM, however, has the attraction that it uses a process that will now be broadly familiar to readers of this manual and, because of its structured and quantitative form, allows aspects of problems to be explored that other mapping procedures do not.