PFHS and Assessment Considerations

From the empirical studies in this thesis and within the current literature, there is some evidence that demonstrates PFHS as a valid and reliable approach to assessing clinical

competence. However, as with the discussion in the last section regarding suitability of PFHS for training, these findings do not automatically warrant its use in assessment of clinical competence. In order to determine its role in assessment of competence a key factor that needs to be considered is its educational utility. A commonly described model for measuring assessment utility in medical education is described by Van de Vleuten (1996) as follows:

Utility = educational impact x reliability x validity x feasibility x acceptability

Using this model, it can be seen that utility is dependant on a number of contributing factors. Although investigating the utility of PFHS was not a key research question in this thesis, some of the findings in this thesis do provide support for its use in terms of providing evidence of validity, reliability, and acceptability. However, educational impact and

feasibility remain to be determined. Of note, PFHS inherently has higher costs in comparison to more conventional PTT-based simulations and the question of feasibility was raised by a number of clinicians interviewed in Chapter 5 regarding the skin laceration management PFHS scenarios. At the same time, PFHS is probably more feasible for assessing clinical competence for some clinical scenarios, particularly those that are rare or involve invasive elements in comparison to workplace-based assessments.

With respect to educational impact, as mentioned in Chapter 1, it is widely acknowledged that assessment has the potential to drive learning. Learners tend to learn what is set in an assessment criteria to pass an assessment (Wormald et al., 2009). Feedback from an assessment can also provide learners with insight into their strengths and weaknesses. However, aside from the positive effects on learning, it has also been demonstrated that assessments can give rise to negative and unintentional learning effects. For example, a study by Rudland et al (2008) investigating the effect of OSCE on learning strategies

demonstrated that students elected to concentrate on studying checklists and practicing on their flatmates, rather than increasing workplace based training. A key reason for this was that the OSCE stations in this specific study were noted to be inauthentic to real clinical practice and instead encouraged students to work on exam techniques such as being able to complete a simulated clinical task comprehensively within the time constraints of an OSCE examination. The clinical workplace was therefore not perceived by students to be the most efficient environment to prepare for OSCEs. These findings are however not entirely

negative. The OSCE still exerted a strong effect of motivating students to learn. However, the fact that it draws students away from obtaining real clinical experience is worrying in that it may not encourage students to develop some of the less explicit (i.e. within the “hidden curriculum”), but equally important skills for real professional practice. Taking these into consideration, in theory, PFHS and contextualised simulation therefore has the

potential to drive learners to develop their clinical competence by being authentic to clinical practice, though the true effect of this has yet to be demonstrated empirically.

Another key property of PFHS, which does not come into the above equation for assessment utility, is its ability to increase authenticity. As demonstrated in Chapter 5, clinicians

generally perceived PFHS-based assessments to be authentic to real clinical practice, though there remains the question of what value this may have. A possible answer is its role in assessment validity. To understand this, we need to first discuss the nature of validity and authenticity and their relationship. Many proponents of the authentic assessment paradigm argue that, by nature of approximating real life practice, it infers strong validity (Archbald and Newmann, 1988; Wiggins, 1993). However, the terms validity and authenticity are not synonymous and should not be confused with one another. Validity relates to whether the assessment measures what it intends to measure. It is therefore a relative measure in

relation to a defined assessment criterion. For example, as discussed in Chapter 2, taking the assessment of competence of suturing skills, if the assessment purpose and criteria is to assess basic psychomotor skills, then assessing someone’s ability to suture a banana may be equally valid to suturing a highly realistic skin simulator (Kyaw Tun et al., 2011). Authenticity of an assessment, on the other hand, is referenced to real life clinical practice. If the criteria of an assessment are referenced to real life clinical practice, then, by increasing authenticity of a simulation-based assessment, validity should also be increased. In this respect, the use of PFHS should be dependant on the purpose of the assessment, which, if is to assess real life clinical competence, then may be beneficial in terms of validity. This is supported to an extent by some of the study results in this thesis. In Chapter 6, the PFHS were designed to match the real patients in terms of signs, symptoms and patient interaction, which may have resulted in the strong correlation in the performance ratings seen between the two types of OSCE stations.

There may be further value of increasing authenticity beyond content validity, i.e.

comprehensiveness of an assessment of the different aspects of a clinical competence for a given clinical skill. If the sole purpose of the assessment was to ensure that there was adequate sampling of the individual component skills required for clinical competence, one could argue that a series of more focused assessments to measure each component

individually should suffice. However, the combined simultaneous assessment of the various components of clinical competence may produce effects that are different than when just assessing each component skill alone in series. This is an area that requires further empirical research.

There is a finding in this thesis, which may support the notion that there is value of

authenticity beyond increasing validity. In the study in Chapter 5, clinicians stated that in the PFHS scenarios, by allowing them to simultaneously exhibit various component skills as they would in real clinical practice, the simulation allowed them to behave more in line with how they would do in usual clinical practice. The issue of trying to capture authentic clinical behaviour in assessment may be an important one, particularly if trying to determine aspects of performance such as professionalism. It is also worth noting that the issue of

due to the inherent difficulty in trying to define and assess this aspect of performance (Riley and Kumar, 2012).

Of note, some clinicians commented on the difficulty in performing well or producing errors when performing in the PTT-based simulation due to its relatively inauthentic nature. This issue of the inauthenticity of some simulations giving rise to potentially negative

performance has been raised previously (Fidment, 2012). This inauthenticity of the

simulation-based assessment may require assesses to adopt strategies or “exam technique” not directly related to clinical competence in order to do well. This is potentially a wider issue not limited to simulation but assessments in general. From my observations as a teacher and examiner, it is not uncommon for assessments to require assesses to be able to exhibit a degree of “exam technique” in order to pass, but this should not be the emphasis of the test. Instead, an assessment should ideally be designed to account for or minimise requirements for exam technique, such that assesses are free to demonstrate their true ability. In this respect, increasing authenticity to allow clinicians to behave as they may do in real clinical practice may be of benefit.

Finally, as discussed in Chapter 1, the authenticity of an assessment should provide value beyond the classroom, which in the case of healthcare education is lifelong professional clinical practice. This requires an assessment to not only reflect a real life activity, but also promote and foster desired values that beyond the actual activity itself. “Contextualising” simulation such as with this technique of PFHS, which appears to bring about more

meaningfulness, allowing users to relate to their professional identity, may be one way of achieving this. In this respect, it is aligned with some of the principles of authentic education discussed in Chapter 1.

Taking the issues of utility and educational impact of assessment discussed above into consideration, there is a general consensus that no single assessment method is always superior to another, and that a multimodal, multidimensional approach is required (Epstein and Hundert, 2002). The use of PFHS, as with other forms of assessment, should be aligned with the desired learning and curriculum objectives. Furthermore, to understand the role of

PFHS-based assessment, it needs to be considered in light not only of its own individual utility, but also in relation to other available assessment methods currently used.