Feedback

In the literature two main types of feedback were tested for their usefulness: outcome feedback (OFB) and cognitive feedback (CFB). Balzer, Doherty and O'Connor (1989) present a useful diagram to illustrate the components possible in feedback information (Fig 3).

OFB involved giving the ‘correct’ answer (‘criterion’ judgement, Ye) in order that a subject could reattempt the task to see if the outcome knowledge helped them to move their own answer (Ys) nearer to the correct one. Correct answers were not available for all decisions, but for some tasks the ‘criterion’ judgement could be found from the task environment. For example, in order to give medical students OFB on how to predict the risk of cardiovascular death, the correct levels of risk could be calculated by using a published logistic regression equation derived from the Framington Heart Study (Tape et al. 1992).

The other type of feedback most commonly investigated was cognitive feedback. This involved giving information on three types of relationships: relationships between the cues and the criterion judgements made (task information), the cues and the subject’s judgements (cognitive information), and the criterion judgement’s with the subject’s judgements (functional validity information).

Task information related to information about the task system (i.e. the environment). Three types of task information were possible. rie is the relationship between the cue (X) and the criterion (Ye) thereby representing how the cue is used. This may include the function form (linear or otherwise). The task predictability, defined as the degree to which the criterion could be predicted given knowledge of the cues was presented in the multiple correlation indices (Re). Intercue correlations were identified with rij. Task information could also be provided on the standard deviations of the criterion across the profiles (SDYe) (Balzer and Sulsky 1992).

Cognitive information provided information about the decision makers own cognitive system. The individual’s judgements on each profile were Ys. The relationship between the cue (X) and the criterion (Ys) was represented both by the cue weight (ris) and the function form (linear or otherwise). The judgement consistency, or what may be more comprehensively termed as cognitive control (Hammond et al. 1975), was defined as the degree to which the criterion could be predicted given knowledge of the cues. This was presented in the multiple correlation indices (Rs). Rs would be influenced by two factors: the degree to which the individual had

used a linear model and how consistently they had applied their model (Hammond et al. 1975). Cognitive information could also be provided on the standard deviations of the criterion across the profiles (SDYs)(Balzer and Sulsky 1992).

Functional validity information linked the task system and the cognitive system. The achievement index (ra) was the correlation between the actual criterions in the task system and the judgements. This essentially showed how close the individual was able to judge correctly. The correlations between the predictions of the linear model of the environment and the linear model of the judge was known as G. The correlation between the residuals from the predictions of these two models was known as C.

Where the environment did not provide the possibility of task information, judgement policies would first have to be ‘captured’ so that they could be used as feedback. The judgements could then be used as feedback for subsequent trials. In these ‘single system’ scenarios, where polices where had to be ‘captured’, cognitive information was otherwise the only feedback information available. An example of policy capturing for use as feedback can be seen in one of Kirwan’s studies of rheumatologists. The rheumatologists were asked to agree on the entry criteria for patients to be used in a clinical trial (Kirwan et al. 1983). Each judged the suitability of 90 paper patients. The relative importance (weights) they attached to each of the cues in the profile was identified by regressing their ‘suitability for entry’ decisions on to the paper patient profiles they had examined. These policies were then used to examine the effects of cognitive feedback by comparing their agreement scores before and after they were given their own policies. Their agreement improved with cognitive feedback. This was shown by the increase in their correlation scores between their two sets of ratings: r = 0.63 increased to r = 0.76. Prior to having been given their own weighting policies, the rheumatologists were given an hour to discuss their decisions with each other. It was found that their agreement did not improve through their discussions. The discussions provided them with each other’s decisions i.e. ‘outcome’ feedback but r =0.63 only increased to r =0.64. The benefits of cognitive feedback, over outcome feedback have been found to be a common research result (Hammond and Summers 1972). It has even been found that cognitive feedback on its own has been more beneficial than giving combined cognitive and outcome feedback. This rather surprising

finding was thought to be due to the fact that judges preferred to ‘chase an error’ (reinforced by outcome feedback) rather than make use of the information available (cognitive feedback) to increase their knowledge (Doherty and Balzer 1988).

In a few studies where outcome feedback has been purported as being more effective than cognitive feedback, there appears to have been a differing view as to what defined cognitive feedback (Balzer et al. 1989). The confusion seemed to have been due to the fact that feedback of the criterion judgements (Ye), although defined as outcome feedback, is actually also part of the task information of cognitive feedback. The presentation of Ye after producing Ys was therefore only a weak form of task information (Doherty and Balzer 1988). Some studies had chosen, incorrectly, only to give the cognitive information part of cognitive feedback, (as opposed to including the task information as well), viewing task information as more closely associated with outcome feedback. Two studies that identified outcome feedback as superior to cognitive feedback did not recognise task information as part of cognitive feedback (Balzer et al. 1989; Tape et al. 1992). For example one of these studies gave their control group the task information of ideal cue weights (which improved their scores) and gave the ‘cognitive feedback’ group cognitive information only (own use of weights) (Tape et al. 1992). As task information, which is normally also part of cognitive feedback, had already been given to the control group, it was not surprising that the cognitive feedback group’s scores did not improve over those of the control group (or as much as the ‘outcome feedback’ group). In fact, subsequent research has shown that it is actually the task information, rather than the cognitive information or the functional validity information that is the valuable part of cognitive feedback (Balzer et al. 1989; Balzer and Sulsky 1992). How the terms have been defined has certainly varied between investigators. It is necessary to ensure the types of feedback information are defined in order for various studies to be critiqued on an equal footing.

Balzer and colleagues had conducted an extensive review of the literature in 1989, but in 1992 they also conducted their own very thorough investigation of the effect of the different types of feedback on performance. Undergraduate students (N=133) were used as participants in a laboratory experiment. Balzer and colleagues set out to test the effects on performance of using all combinations of the major cognitive feedback components ie task information,

cognitive information and functional validity information. They also incorporated a control group who received no feedback. Participants had to accurately predict the number of wins for baseball teams. Effects on Rs, ra, G and C were used to examine effectiveness of the differing conditions. Self-report was also used to collect subjective views of the helpfulness and understandability of the feedback. Task information, on its own, was found to be the most effective feedback for producing significant changes in performance. This was identifiable through changes in ra (achievement) and G (knowledge). Those participants who received cognitive feedback did not differ significantly in their results from those who received no feedback. When task information was combined with other types of feedback, no additional improvements were made compared to using task information alone. Participants did not report any particular type of feedback as being more or less useful then any other. Participants were therefore not aware of how beneficial the feedback was to them.

Conclusions that arise from this research indicate that task information is the most valuable type of information needed to promote learning. However as task characteristics can affect how cues are used, other tasks with different characteristics may yield different results. Balzer’s task had used cues with low intercue correlations and linear cue relationships (Balzer and Sulsky 1992). Tasks using high intercue correlations and configural cue relationships may require differing feedback to maximise their use.

In conclusion, the task information component of cognitive feedback has been found to be the most useful information for improving performance. This is the most beneficial information for judgements that use cues with low intercue correlations and with linear relationships to the criterion. ra (achievement) and G (knowledge) can be expected to improve as an indication of improved performance. The effect of improved performance on other lens model indices, such as C, is less well understood. Research is needed to develop information that can be used as task information. Policy capturing studies with clinicians are needed to provide this information. The aspiration to use feedback to train clinical students can then be realised (Chaput de Saintonge and Hattersley 1985; Tape et al. 1992). In the clinical field where practitioners must make “multiple-cue judgements of some distal, imperfectly knowable

criterion” (Balzer et al. 1989) p. 430, task information has been viewed as having ‘extraordinary potential’.