Workplace based assessments (WBA)

political and professional accountability. Essential surgical qualities including history taking, clinical examination, leadership and specialty related expert knowledge have traditionally been assessed at board examinations and interviews. However, in the arena of arguably the most important attributes of manual dexterity and technical skill, objective assessment has lagged behind. Workplace based assessments were introduced to bridge this apparent gap and to improve the efficiency of learning by informing more focused educational activity (Beard et al. 2011). Direct trainer feedback of observed clinical interactions and technical skills has been shown to be beneficial to learning (Hattie and Timperley 2007; Beard 2008).

Essential to the introduction of new assessment tools however is the need to ensure that it is reliable, valid, feasible, and has educational impact. There has been some resistance to the uptake of such tools, with a number

educationalists and clinicians stating that the justification for their

implementation has been based on weak evidence (Pereira and Dean 2013). Early in their introduction into UK surgical training, WBAs were singled out amongst trainees as being a source of dissatisfaction with one of the earliest surveys reporting up to half of the 539 respondents describing WBAs as ‘poor’ (Pereira and Dean 2009). A concerted nationwide effort ensued, to educate trainees about the intentional formative use and value of WBAs, which resulted in a significant improvement in satisfaction rates (Marriott et al. 2011; Pereira and Dean 2013; Hawkins et al. 2014; Hunter et al. 2015; Phillips et al. 2015).

A description of the assessment tools in practice and their evidence base is given below:

Case Based Discussions (CBD)

The Case Based Discussion was first designed to assess clinical judgment, professionalism knowledge and decision making in relation to a case in which a trainee has been directly involved. The basic premise involves an in depth two- way discussion with a clinical supervisor exploring the trainee’s reasoning around the management of a patient. The CBD encourages trainees to reflect on learning and identify any potential development needs (Phillips et al. 2015). A recent study by Phillips et al. demonstrated that surgical trainees valued CBDs, their use encouraging higher thinking and reflection however concerns existed about their potential use as a tick box exercise (Phillips et al. 2015). There is little research on the validity of CBD in the surgical workplace setting, however research in medicine has shown its positive educational impact (Johnson et al. 2011). Within Otolaryngology trainees, CBD was shown to be reliable with good construct validity (Awad et al. 2015a).

Mini Clinical Evaluation Exercise (Mini-CEX)

The Mini-CEX mainly assesses clinical and professional skills in a work setting, with the trainer observing a trainee’s interaction with a patient. The area of emphasis may include history, examination, diagnosis, investigation or management, with communication being assessed throughout.

A meta-analysis of studies looking at the use of the Mini-CEX within

postgraduate medical training revealed evidence to support good construct- validity and hence its use within medical training (Al Ansari et al. 2013). Within surgery, the Mini-CEX has been found to be of more use in earlier years of training with poor discriminatory value amongst trainees in their later years of

assessments early in their training (Awad et al. 2015a).

Direct Observation of Procedural Skills (DOPS)

The DOPS is the direct observation of a basic diagnostic or interventional skill performed by a trainee, and is used to assess technical, operative and

professional skills. This tends to be reserved for use for trainees in their early core years, with Procedural Based Assessments (PBA) used for Higher Surgical Trainees in more complex procedures and operations.

DOPs, although shown to accurately differentiate between junior and senior trainees, does not clearly demonstrate progress above ST3 level, due to the simplicity of the procedures (Awad et al. 2014). It is for this reason that they are restricted for use in foundation and core surgical training. A systematic review of the use of DOPs in medical education revealed generally good reliability and acceptability, however research on validity and educational impact was lacking (Naeem 2013).

Procedure Based Assessment (PBA)

As with the DOPS tool, Procedural Based Assessments examine a trainee’s technical, operative and professional skills in a range of routine surgical procedures. Derived from the Orthopaedic Competence Assessment Project (OCAP), it was developed to assess orthopaedic surgeons; the PBA is procedure specific and involves direct observation of an index operation. Assessment is broken down into two components (i) overall ability to perform the procedure, and (ii) a series of competencies specific and necessary to undertake the operation (Pitts et al. 2005). The highest rating is commensurate with the ability to be able to undertake the operation at a level equivalent to that

of a consultant surgeon. This form of assessment is often used instead of the DOPS for Higher Surgical Trainees.

A number of studies have shown PBA to be a well-validated tool (Beard et al. 2005; Pitts and Rowley 2009; Beard et al. 2011; Marriott et al. 2011; Awad et al. 2015b). A large study performed in 6 surgical specialties within Sheffield

teaching hospitals demonstrated clear construct and content validity, as well as excellent reliability and positive overall satisfaction amongst users (Beard et al. 2011; Marriott et al. 2011). Importantly this study demonstrated that achieving PBA reliability did not require rigorous clinical supervisor training due to the intuitive binary scoring system and descriptive anchor statements provided for overall assessment of competence. Another more recent large UK study of over 3000 ENT PBAs similarly demonstrated excellent reliability and validity (Awad et al. 2015b).

Mini Peer Assessment Tool (Mini-Pat)

This is a 360-degree multisource feedback that largely covers a trainee’s professionalism within the multidisciplinary team. The Mini-PAT requires a trainee to nominate a minimum of 8 raters from allied professionals within the MDT. Assessment forms are mapped against the GMC’s standards of Good Medical Practice, with feedback anonymised. This is normally undertaken annually and has been found to be a feasible, reliable and valid tool in assessing non-technical competencies in surgery. A meta-analysis revealed that professionalism, communication and interpersonal skills could be reliably assessed using this tool (Al Khalifa et al. 2013).