A small sample of studies that have used methodologies similar to that in the current study to analyse relationships between scores in selection and scores in assessments is presented here. Other representative studies are referred to throughout the Literature Review. Typically, such studies assess the extent to which performance in one or more aspects of selection predict subsequent performance. Systematic reviews by Roberts et al (2017) and by Patterson et al (2016) emphasise that reviews of individual selection tools can be limited, inconclusive and contradictory. These researchers highlight gaps in the evidence for the effectiveness of combinations of selection instruments. This study evaluated the performance of individual selection instruments and of the ‘Total selection score’, representing the performance of combined selection instruments.
Carmichael et al. (2005), in their study of orthopaedic surgical trainees’ performance in selection activities and examinations, demonstrated that comparing scores in these instruments revealed the types and the extent of relationships between performance in these activities. Eva et al. (2004) and Eva, Reiter, Trinh, Wasi, Rosenfeld & Norman (2009) similarly used multiple mini-interview scores and subsequent assessment scores to evaluate the capacity of the multiple mini-interview to predict medical students’ performance.
In their 2002 study of a dental training program, Sandow, Jones, Peek, Courts and Watson (2002) found a statistically significant correlation between low selection scores and poor performance. Sandow et al.’s use of multivariate correlation to analyse the relationships between many admission criteria (including academic scores, interview scores and a dental admissions test) with dental school performance measures is consistent with the methodology used in the current study.
In a comparable study, Wilkinson, Zhang, Byrne, Luke, Ozolins, Parker and Peterson (2008) assessed how well prior academic performance, admission tests and interviews predicted academic performance in the medical program at the University of Queensland. Wilkinson et al. calculated the correlation between student academic performance and selection criteria for three cohorts (2001–2003 entry years), using Spearman’s correlation coefficient. The study used multiple linear regression to compare outcome variables: clinical exams, written exams, and ethics exams, against predictor variables: academic GPA, the Graduate Australian Medical School Admissions Test (GAMSAT) and an interview. The researchers found that the selection criteria used by the University of Queensland predicted about 20 percent of examination
performance. Wilkinson et al. (2008) reported that the predictive power of interview performance was higher for academic performance at the end than at the start of the medical program and cautioned that “most variation in academic performance is not explained by selection criteria” (p. 351). The researchers propose that their study was strengthened by examining a range of selection criteria in association with each other, not in isolation.
Summary to the Literature Review
From an examination of prevailing research, I contend that selection to specialty medical training is an inexact pursuit which, despite high-stakes consequences, has yet to be reliably standardised. Selectors aspire to admit individuals who will make proficient trainees and practitioners (Carroll et al., 2009; Cleland et al., 2012; Elfenbein et al., 2015; Gallagher et al., 2008; Martin, 1996; Makdisi et al., 2011; Thordarson, et al., 2007); however, selection
future. Multiple facets of selection have been studied to elucidate the nature of their contributions and their impact on selection outcomes. Studies predominantly focus on functional aspects of selection and implications for practice. As implementation of selection instruments and processes appears to be highly context–specific, the findings from these studies frequently differ and may not be directly comparable. Some fundamental principles, such as aspiring to fairness may be so pervasive that they are seldom elucidated; in contrast, there is strong interest, and less agreement, regarding key issues such as the goals of selection and the relative importance of designated knowledge, skills or attributes sought in candidates and how to assess these.
Candidates’ perceived merit for selection into surgical training is measured and assessed through their performance in selection instruments, but may also take account of social, cultural, racial or other attributes. There is a tension between the importance of diversity, the need to address historic inequities in the composition of the profession, and the notion of merit-based selection. Compensation for disadvantage or inequity in preparatory experiences may be applied in addition to performance in selection instruments. Judgements regarding the intent and implementation of compensatory measures must be communicated to participants. Review of the effects and outcomes of compensatory measures will feed into the cycle of selection development and implementation activities.
Through the literature, I have identified that non-cognitive skills as well as technical motor skill acquisition and proficiency are important to optimal surgical practice and may develop during training. Some trainees may demonstrate great technical aptitude; others may take longer to learn technical skills, but may demonstrate early proficiency in cognitive or other non-technical skills. Trainees will enter surgical training with varying degrees of aptitude across the range of identified competencies and will develop skills at different rates in response to many stimuli and experiences. The current study did not address aptitude beyond that tested in the RACS selection instruments.
One proposal to maximise reliability of selection is to introduce “more robust work-based assessment tools” to assess performance of candidates prior to selection (Collins, 2007, p. 9). Using results from structured WBAs in selection has the potential to enrich the information that candidates bring to selection by adding ‘authenticity’. It may enhance reliability by reducing the time lag between performance and assessment and may reduce pressure on assessors to complete and return reports to selection administrators. However, such assessment would be reliant on assessors cooperating, regardless of whether they had an affiliation to a specialty training program. To optimise this approach, assessment tasks and instruments would need to be predetermined; then it could draw on a model in which candidates instigate assessments to build longitudinal portfolios of their achievements. Such an approach could improve the validity and reliability of assessments and be cost and time efficient for specialty colleges. Further study would be required to determine the extent to which this approach would assist selectors to differentiate and rank candidates.
In this chapter I have described four conceptual frameworks that contribute to our understanding of selection to surgical training, offered insights into key issues that impact on surgical training and selection, analysed general and specific attributes of selection and assessment instruments and protocols, examined eight prevalent selection instruments, and presented a sample of studies that have used methodologies approximating that in the current study. It is important to note the interplay of multiple influences and determinants that affect protocols and outcomes in selection and assessment. It is also relevant that developments and studies in this field are ongoing, incrementally adding to our identification and understanding of practices to enhance selection outcomes.
The primary purpose of this literature review was to create an information base for investigation of the RACS GS selection instruments. It has presented a set of conceptual frameworks, a broad spectrum of selection issues and attributes and the outcomes of relevant research to support the study. This dissertation will integrate the outcomes of the literature review into an evaluation of selection practices for surgical training in Australia and New
Zealand. Without reliable and valid measures of identified attributes, selection to surgical training will continue to be an imprecise endeavour with uncertain outcomes.