performance matr
5.6 Step 4: Assessing performance levels (with scoring) The first consideration in setting up consistent numerical scales for the
assessment of criteria is to ensure that the sense of direction is the same in all cases, so that (usually) better levels of performance lead to higher value scores. This may mean a reversal of the natural units. For example, access 29 For example, Bazerman, M H (1998) Judgement in Managerial Decision Making John Wiley, New York, Fourth Edition.
to a facility might be recorded in terms of distance to the nearest public transport, where the natural scale of measurement (distance) associates a low number with a good performance.
It is conventional to allot a value score to each criterion between 0 and 100 on an interval scale. The advantage of an interval scale is that differences in scores have consistency within each criterion, although it does not allow a conclusion that a score of 80 represents a performance which on any absolute standard is five times as good as a score of 16 (which would require a ratio scale of measurement). The ‘ruler’ which the scoring scale represents is good only within the confines of this particular MCA. However, when combined with appropriately derived importance weights for the criteria, the use of an interval scale measurement does permit a full MCA to be pursued. The first step in establishing an interval scale for a criterion is to define the levels of performance corresponding to any two reference points on the scale, and usually the extreme scores of 0 and 100 would be used. One possibility (global scaling) is to assign a score of 0 to represent the worst level of performance that is likely to encountered in a decision problem of the general type currently being addressed, and 100 to represent the best level. Another option (local scaling) associates 0 with the performance level of the option in the currently considered set of options which performs least well and 100 with that which performs best.
The choice between local and global should make no difference to the ranking of options. An advantage of global scaling is that it more easily accommodates new options at a later stage if these record performances that lie outside those of the original set. However it has the disadvantages of requiring extra, not necessarily helpful judgements in defining the extremes of the scale and, as will be seen in the next chapter, it lends itself less easily than local scaling to the construction of relative weights for the different criteria.
Once the end points are established for each criterion, there are three ways in which scores may be established for the options.
The first of these uses the idea of a value function to translate a measure of achievement on the criterion concerned into a value score on the 0 – 100 scale. For example, if one criterion corresponds to number of regional full- time jobs created and the minimum likely level is judged to be 200 and the maximum 1,000, then a simple graph allows conversion from the natural scale of measurement to the 0 – 100 range required for the MCA. This is shown in Figure 5.2.
Figure 5.2 Regional jobs
For any option, its score on the regional job creation criterion is assessed simply by reading off the vertical axis the score corresponding to the number of jobs created, as measured on the horizontal axis. Thus an option that creates 600 jobs, say, scores 50.
Where higher measurements on the scale of natural units correspond to worse rather than better performance, the slope of the function mapping achievement level on to 0–100 score is simply reversed, as in Figure 5.3.
Figure 5.3 Distance to public transport
The value functions used in many MCA applications can for practical purposes be assumed to be linear. However, on some occasions it may be desirable to use a non-linear function. For example, it is well known that human reaction to changes in noise levels measured on a decibel scale is non-linear. Alternatively, there are sometimes thresholds of achievement above which further increments are not greatly appreciated. For example, in valuing office area, it may well be that increments above the absolute minimum initially lead to substantially increased judgements of value on room size, but after an acceptable amount of space is available, further marginal increments are valued much less highly. This is illustrated in Figure 5.4. In this case, the judgement is that, once the area reaches about 250 square metres, further increments add less value.
The second approach to scoring performance on an interval scale is direct rating. This is used when a commonly agreed scale of measurement for the criterion in question does not exist, or where there is not the time nor the resources to undertake the measurement. Direct rating uses the judgement of an expert simply to associate a number in the 0–100 range with the value of each option on that criterion.
Because these scores are being assessed on an interval scale of measurement, relationships between the differences in options’ scores do have meaning and it is important to check that the judgements being made are consistent in this respect. Specifically, a difference of (say) 20 points should reflect an improvement in assessed value which is exactly half that measured by a difference of 40 points.
Figure 5.4 Diminishing returns to area
The use of direct rating judgements in MCA can pose some problems of consistency in circumstances where the procedure is to be applied by different people, eg, where some decision making responsibility is delegated to regional offices. The simplest way to encourage consistency is to provide a set of examples or scenarios with suggested scores associated with them. Another issue to bear in mind with direct rating is that sometimes those with the most appropriate expertise to make the judgements may also have a stake in the outcome of the decision. Where this is the case, there is always the danger that their rating judgements may (perhaps unconsciously) be influenced by factors other than simply the performance of the options on the criterion being assessed. Ideally, such judgements should come from individuals who are both expert and disinterested. If this cannot be managed, then it is important be aware of the possibility of some bias creeping in and, for example, to apply sensitivity testing to the scores at a later stage as a means of checking the robustness of the outcome of the analysis.
A third approach to scoring the value of options30 on a criterion is to
approach the issue indirectly, by eliciting from the decision maker a series of verbal pairwise assessments expressing a judgement of the performance 30 Each of the methods mentioned here can also be used to establish the relative weights to be given to criteria in the full
of each option relative to each of the others. The Analytic Hierarchy Process (AHP) does this (Chapter 4 and Appendix 5). Alternatives are to apply REMBRANDT (Appendix 5) or MACBETH (see, eg, Bana e Costa and
Vansnick, 1997, Bana e Costa et al, 1999).31
To illustrate, the MACBETH procedure asks decision makers to assess the attractiveness difference between each pair of options as one of:
C1 very weak difference
C2 weak difference
C3 moderate difference
C4 strong difference
C5 very strong difference
C6 extreme difference
Once all the required pairwise comparisons are made, a series of four computer programmes processes these data to calculate a set of scores for the options, on a 0–100 scale, which are mutually consistent with the full set of stated pairwise judgements. If, as can happen, there are inconsistencies within the judgements, such that a compatible set of scores cannot be computed from them, the programmes guide the decision maker through steps to amend the inputs until consistent scores are obtained.
Following one or other of these procedures through one by one for each of the criteria provides the full set of value scores on which any compensatory MCA must be based. Applications which apply formal numerical comparisons to options are covered in Chapter 6.
31 Bana e Costa, C.A. and Vansnick, J.C. Applications of the MACBETH approach in the framework of an additive aggregation model, Journal of Multi-criteria Decision Analysis, vol. 6, pp. 107–14, 1997, and Bana e Costa, C.A., Ensslin, L., Correa, E.C. and Vansnick, J.C. Decision support systems in action: integrated application in a multi-criteria decision aid process, European