Assessment of performance in simulation

Facilitated debriefing in simulation implies at least an informal external assessment of performance. The facilitator will usually make some judgement of the learner level of performance and relate this

to either established standards (criterion reference) or the performance of peers (normative referenced)(59,88). Frequently, this is not formally recorded or carried forward beyond the reflection of the learner however it has been proposed that workplace based assessments may be recorded based on simulation performance, especially for simulated scenarios which are uncommon and that the learner may not experience in clinical practice. This is a divisive subject for simulation educators. The psychological safety of the learner is a key consideration here; one of the defining features of simulation is that no harm can occur to patients when mistakes are made, and hence learners may try new approaches. If an assessment is being made, this may restrict the freedom of the learner to do so and also places the educator in the role of assessor(89–95). This may impact upon the interaction between educator and learner at the time of debriefing, limiting the discussion and hence the potential learning from the experience. On the other hand, however, are the duties placed upon the educator as a healthcare professional. If conduct which could be harmful to patients were observed during simulation and no satisfactory resolution is reached during debriefing, there may be an onus on the educator to act upon this to prevent potential patient harm. This may take the form of further assessment or simulation sessions, or alternatively an attempt at direct correlation between simulation performance and clinical performance, with liaison with clinical supervisors.

One of the challenges with assessment in simulation is the validity of tool used. The Mini-CEX assessment tool has been validated for use in clinical environments but even then, a significant subjectivity from raters can be shown(96,97). There has been no validation for its use in simulation, although this is common practice. The OSATS assessment of surgical performance is amongst the few which have been validated for use in simulation (98–102). Many of these are objective skill based assessments, which assess metrics which are either direct measurements or can be scored pseudo-objectively by expert observers. No other workplace based assessments which are in common use have been validated for use in simulated environments.

There is increasing interest in the use of simulation for summative assessment and

certification(93,103,104). There are many potential advantages of this, including reproducibility of scenarios and the ability to tailor simulation scenarios directly to the factors being assessed. Construct validity of the assessment of performance in simulation relies on the assumption that learner performance in simulated environments is truly reflective of real clinical practice. The perfect scenario in assessment terms would be where the participant could not tell the difference between the simulation and real life. An example of this is the use of unannounced simulated patients used in general practice. In reality, although great care is taken to create an accurate analogue, there may still be discrepancies such as location, equipment, paperwork, time and staffing, which may alter learner response. Learners may not attach the same urgency to actions, or situational changes, may engage in riskier or untried behaviour, or may simply fail to consider the activity serious or relevant to their clinical practice and disengage from the process. This is perhaps less likely with

preannounced assessments but little evidence exists on the subject. In North America, clinicians can already opt to have part of their professional certification assessments in simulation and with the development of robust, simulation specific assessment tools, this is likely to increase(105).

1.7.1 The use of video recording and its use in debriefing

In addition to the reproducibility of scenarios during simulation, another advantage is the ability to record video and audio of the events that occur and play this back in the debriefing. This may allow the demonstration of acts such as non-verbal communication behaviours which can be difficult to otherwise debrief. The cognitive overload that occurs during crises can distort the recollection of events that happened, and evidence from critical scenario debriefing suggests that details and timescales of events are often poorly recalled. It must be considered, however, that the presence of video recording equipment itself may alter the response of the learner. They may experience unease when simulations are recorded or observed by their peers, which may in turn influence

oral debriefing alone; and video recording poses the technical challenges of equipment usage and storage(106–110). The use of video playback during the debrief is itself a skilled process; simply playing back entire scenarios whilst discussing events over the top of it is unlikely to be a useful learning exercise however carefully chosen snippets can help illustrate relevant points, especially where there is discord between learner and educator. In some cases, it is possible to provide the learner with access to the footage to take away with them for review at their convenience but this should not replace debriefing by an educator.