What supervisors value
Findings of interviews with supervisors—Phase 1
The aim of this phase of the research was to gain insights into what supervisors value in veterinary student performance. As conceptualised in the framework presented in Chapter 2, the picture supervisors develop and their expectations of student performance form early steps in the process of judging performance. How these are translated into scores on in- training evaluation and how they align with the intentions of the assessment can then be investigated in later phases of this research, to inform our understanding of the process of judging the competency of veterinary students, and the validity of the scores. Individual interviews were therefore used to explore supervisors’ depictions of students performing at different levels of competency. The design mirrored research that has been conducted in medical education and social work education (Bogo et al., 2006; Ginsburg et al., 2010). These previous studies gave insight into the range of aspects considered by supervisors, and how these were weighted and balanced for students of differing performance levels in these disciplines. They revealed that not only did what was considered differ for different students, but that it was not always well aligned with the intentions of the assessment. Therefore investigating what veterinary supervisors value is an important first step in investigating how well the scores on this in-training evaluation instrument capture the aspects of student performance we intend to assess.
Research method
Interviews
Semi-structured interviews were used to collect data for this phase of the research. Interviews were an appropriate method for collecting information about people’s views and perspectives
aspects they discussed. The semi-structured format enabled me to phrase and rephrase as necessary to help understanding of the questions. It enabled me to redirect, clarify, or seek more detail, if needed, to help my understanding and so that I gained as full a picture of their views as possible. Individual interviews were most suitable because the supervisors were likely to feel comfortable expressing their views one-on-one and because the purpose was to understand individual views in-depth.
The interview protocol (Appendix A, page 251) was adapted with permission from that used by Ginsburg et al. (2010) with supervisors of medical students. It approached the research question indirectly by asking supervisors to describe students they had previously encountered that they considered to have been excellent, weak, and marginal on placements. This was a way of getting a sense of what supervisors valued, without asking them directly about it. It also separated the context of the questions from the context of the in-training evaluation instrument, and from any competency frameworks. The aim was to tap into what supervisors considered and valued, rather than what they thought they were supposed to consider and value. This meant the themes arising were not externally derived, but those in use.
The wording of questions was changed slightly from the protocol used by Ginsburg et al. (2010) so that it applied to veterinary students. Another change was to ask for descriptions of marginal students, rather than average students. Ginsburg et al. (2010) has asked for descriptions of average students, but found that these did not elicit additional themes. The term “marginal” was chosen instead because the correct categorisation of marginal students is an area of difficulty and it was thought this may elicit valuable descriptions which would help clarify the difference between good students and weak ones. During the interview, participants were asked to describe a particular student they remembered, not an amalgamation of various students. This was to ensure the description was not an idealised— and potentially unrealistic—one, and to capture examples of how supervisors weighted or disregarded aspects of the student performance in forming an opinion of them. After supervisors had given their initial description of what made the student excellent/weak/marginal, I prompted them to clarify aspects of what they had said and to elaborate with examples of what they had observed. I then prompted them to cover any aspects of performance they had not already talked about such as the student’s knowledge, application of knowledge, technical and animal handling skills, communication, and work with others, wherever these were relevant to the placement. They were also prompted to describe
strengths or weaknesses if they had not already described any. A pilot of the adapted interview protocol with two staff members demonstrated that the questions functioned well to elicit descriptions of students in all three categories. The pilot interviews were not included in the analysis.
I conducted the interviews with each participant in person in their own office or workplace and audio-recorded them, with their permission. The interviews were then transcribed verbatim by a research assistant who had signed a confidentiality agreement. Pseudonyms were used to protect the identity of participant supervisors. I verified and corrected the transcripts by listening to the recordings. Each transcript was also verified by the participant who had the opportunity to clarify or correct any part of the transcript. Only a few participants had changes and these were minor, such as typographical errors. Both the transcripts and the recordings were uploaded into NVivo (version 10, QSR International) for analysis. This allowed concurrent analysis of audio and written versions to aid in the capturing of the meaning intended by participants as conveyed by tone of voice, pauses, and non-verbal sounds.
Participants
Interview participants were purposively selected from those supervisors who had been performing in-training evaluations of final year students from Massey University at the time of the study. Only those supervisors from the core (compulsory) placements were included for two reasons. Firstly, supervisors on core placements had supervised large numbers of students over the year, and so had a great deal of experience from which to form their views and draw examples. Secondly, since all students had to participate in the core placements, each supervisor was likely to see a spectrum of students with a range of interests, behaviours, and personalities.
Of the 54 supervisors who supervised core placements, I planned to interview 15 participants because this was a manageable number that had been sufficient to reach saturation of themes in a previous study of a similar design (Ginsburg et al., 2010). In order to capture a range of views, supervisors from across the range of subdisciplines and sites (on-campus or off-campus)
number of supervisors were selected from subdisciplines and sites in which students spent more time during their core placements. Only supervisors who were veterinarians were included because others would not have the same educational and professional experience, and thus would have had different influences on their views. Only supervisors with at least two years of experience supervising students on placement were included, to ensure they had adequate experience on which to draw for the interviews. Lastly, participants were also selected to provide representation from both genders and a range of years of experience. Of the 15 supervisors invited to participate based on these criteria, all agreed to participate. There were four female and 11 male participants. Three were non-academic staff in private practices and 12 were academic staff working on campus. There were four supervisors from the production (farm) animal subdiscipline, two from equine, five from small animal (dog and cat), two from pathology, one from diagnostic imaging and one from anaesthesia.
Thematic analysis procedures
Constant comparison analysis (Leech & Onwuegbuzie, 2007) was used to investigate the interview data. This is the method of choice when research questions are general and overarching (Leech & Onwuegbuzie, 2007). In constant comparison analysis aspects of responses are categorised (coded) and codes are then grouped into themes (Leech & Onwuegbuzie, 2007). The codes used were generated from investigation of the transcripts and recordings (i.e. emergent) in an iterative process. Dual coding was permitted.
Within NVivo, I read and initially coded the transcripts to identify key ideas. These were segments of descriptions that seemed significant either because they were recurrent, were emphasised by a participant, linked with learning outcomes and competency frameworks, or had been mentioned by other authors previously in veterinary science or other disciplines. The linked recordings were used to clarify aspects of the transcripts as necessary to understand the supervisor’s meaning. This initial coding generated 131 key ideas, some of which were overlapping or related. I then sorted the key ideas into groups according to the similarities in the ideas. This resulted in 12 themes which summarised the key ideas. Three framework matrices were then created according to the 12 themes, one for each type of student (excellent, weak, marginal). The framework matrix was a table with the 12 themes as rows and
the participants as columns. In each cell of the matrix, I summarised the theme from the original transcripts. This process allowed me to check that the themes were an appropriate grouping for the key ideas. The matrix framework summaries for each participant were then themselves descriptively summarised to form the descriptions and definitions of the themes. Descriptive information about the way each theme was used, illustrated with excerpts, was also assembled.
Descriptive information about how often a theme was used by supervisors for each type of student they described was also assembled. Together with the emphasis given by supervisors, this was used to give an indication of the relative importance of themes to supervisors. Because some themes were prompted for if not discussed spontaneously by supervisors, the frequency of theme use in the initial spontaneous descriptions was assembled separately from the overall frequency. The use of each theme by supervisors was also categorised as being positive (indicating a strength) or negative (indicating a weakness) or mixed (indicating the theme had both positive and negative aspects) and these were quantified for descriptive analysis. Themes that did not concern the student’s current performance, for example themes that concerned the supervisor, were not included in the analysis of positive and negative comments.
Findings
Themes described by supervisors
Many comments by supervisors were descriptions of the student’s abilities, behaviours, attitudes, and personal characteristics. Nine distinct but interrelated themes could be identified in these comments. Supervisors also went beyond descriptions of the students themselves to comment on the student’s impact, prospects, and the difficulty in judging their competency, resulting in an additional three themes. These are defined in Table 4.1.
both by academic and non-academic supervisors. This demonstrates that the thematic framework developed was useful for capturing the thoughts of supervisors.
Table 4.1: Themes arising from supervisors’ descriptions of excellent, weak, and marginal veterinary students, and their definitions.
Thematic area Theme Definition
Engagement Enthusiasm for and participation in the activities of the placement. Trustworthiness Honesty, reliability, taking responsibility, and ability to discern own limits and act within them, including
perceptions about own abilities Discipline-
specific
knowledge and skills
Knowledge Discipline-specific knowledge Applying
knowledge Problem solving, planning, decision making, report writing, giving presentations Technical skills
and animal handling
Discipline-specific technical and animal handling skills including safe handling of animals
Relating to others
Communication Character and effectiveness of communication and willingness to communicate, listening, body language Social
interactions Social awareness, relationships with others, ability to work with others Personal functioning Management of emotions and personal issues, managing fears, shyness, confidence Caring for animals Showing concern for animals under their care, paying attention to the animals, concern for animal welfare
Other aspects
Impact Effects of the student actions and attitudes on the supervisor or other staff Prospects Predictions about the future career or employment of the student Difficulty in
Figure 4.1: Number of supervisors using each theme in their initial spontaneous descriptions of excellent, marginal, and weak veterinary students.
For example, 13 of 15 supervisors described aspects of engagement in their spontaneous descriptions of excellent students; 12 of 15 described it in marginal students; and 10 of 15 described it in weak students.The category of “other” refers to comments made after description of excellent, weak, and marginal students was complete, when supervisors were asked if there was anything else they would like to add.
Table 4.2: Number of supervisors using each theme in their initial spontaneous descriptions or within all (both spontaneous and elaborated) descriptions of any student.
Thematic area Theme Spontaneous All (spontaneous and elaborated)
Engagement 15 15
Trustworthiness 8 13
Discipline-specific knowledge and skills
Knowledge 11 15*
Applying knowledge 10 15*
Technical & animal 7 14*
Relating to others Communication 12 15*
Social interaction 9 15*
Personal functioning 10 13
Caring for animals 4 9
Other aspects
Impact 7 10
Prospects 3 8
Difficulty in judging 10 14
Note. Asterisks indicate themes that were prompted for if not spontaneously discussed by supervisors and which therefore could be expected to be present in the elaborated descriptions of all supervisors if the theme was
Frequency of student strengths and weaknesses
The initial spontaneous descriptions of excellent students were almost all positive in nature, and of weak students were almost all negative (Table 4.3). Descriptions of marginal students tended also to be negative, but there were a greater proportion of positive and mixed aspects than in weak students. After further elaboration, the balance of positive and negative aspects altered little for excellent and weak students, but the proportion of positive comments about marginal students increased (Table 4.4).
If negative comments were made about an excellent student, supervisors usually balanced them with positive aspects of the same theme. Also, generally, the weaknesses of excellent students tended not to be very negative. For example, Adam mentioned a “slight lack of self- confidence”, a theme that was also picked up on by Gary, and Jean spoke of an excellent student as being “quite a serious, studious looking person”. Ed described an excellent student who lacked “the practical experience of actually doing”. Knowledge that “wasn’t quite up to scratch” was mentioned by Walter and echoed by Ben, Harry, and Todd. There was often a sense that these were only relative weaknesses, for example, Ben explained that “no one knows everything” and didn’t see that “necessarily as a weakness”. Martin and Simon could not identify weaknesses in the excellent students they described.
In weak students, the strengths mentioned were sometimes significant strengths, for example “base knowledge was great”, “nice and friendly”, and “diligent”, but more often were not very strongly positive, for example “affable”, “fine with clients”, and “she was okay”. The strength of two weak students related to improvements made after they failed the placement. Julia and Oscar could not identify strengths in the weak students they described.
Table 4.3: Presence of positive, negative, and mixed themes in descriptions of excellent, weak, and marginal students: spontaneous descriptions.
excellent weak marginal total
positive 44 (85%) 5 (9%) 12 (18%) 61
negative 2 (4%) 48 (84%) 46 (68%) 96
mixed positive and negative 6 (12%) 4 (7%) 10 (15%) 20
total 52 (100%) 57 (100%) 68 (100%) 177
Note. The themes of impact on the supervisor, prospects of the student and difficulty in judging competency are not included in these totals. Percentages are rounded to the nearest integer.
For example, 85% of themes used in descriptions of excellent students were positive, 4% were negative and 12% were a mixture of positive and negative.
Table 4.4: Presence of positive, negative, and mixed themes in descriptions of excellent, weak, and marginal students: spontaneous and elaborated descriptions.
excellent weak marginal total
positive 105 (85%) 15 (13%) 29 (28%) 149
negative 4 (3%) 95 (80%) 60 (59%) 159
mixed positive and negative 15 (12%) 9 (8%) 13 (13%) 37
total 124 (100%) 119 (100%) 102 (100%) 345
Note. The themes of impact on the supervisor, prospects of the student and difficulty in judging competency are not included in these totals. Percentages are rounded to the nearest integer.
Use of themes by supervisors
Engagement
The theme of engagement was used to refer to a student’s enthusiasm for and participation in the activities of the placement which could include specific learning activities such as discussions and presentations, a range of professional activities, or generic work such as fetching equipment or cleaning up. Supervisors sometimes referred to specific tasks but they often spoke more generally. Frequently they described attitudes or dispositions (such as motivation, commitment, laziness) or unobserved behaviours (such as thinking and reflection), complemented by descriptions of observed behaviours (such as initiative, thoroughness, punctuality, doing extra reading, asking relevant questions), or standards of performance (such as work quality).
Engagement was the theme mentioned by more supervisors than any other in their spontaneous descriptions of all types of students (Figure 4.1), and all supervisors used this theme in describing at least one student in their spontaneous descriptions (Table 4.2). This suggested it was very important to all supervisors and a memorable aspect of student performance. For Walter it was the “first thing that made them really excellent”. Adam, Ben, Julia, Harry and Gary all directly stated its importance to them over all other factors. Gary explained that “when it comes down to it, that is all I really care about. I want them to be interested, I want them to be engaged, I want them to be motivated to learn and I want them to enjoy what they’re learning in my rotation.”
Engaged students were ready and eager; they volunteered to do things, asked questions, and seemed to be enjoying themselves. As Adam described, “they’re obviously happy to be there”. Walter, Julia, and Simon also emphasised the eagerness of the engaged students they described, which included being eager to do things that were boring or that other students did not want to do, even at night. For example, Walter described “how unbelievably enthusiastic they were, to do something, it didn’t matter what time, what time of day or place even, they’d be keen to, to go out and do things”. Todd and Harry also described engaged students who looked for things that needed to be done and tried to be helpful.
Even when not actively doing things, students demonstrated their engagement through their body language, making eye contact, and paying attention. They also showed they were actively thinking and reflecting on what was going on by “asking questions that let you know that they were synthesising the information and taking it a step further” (Ben), and as Martin and Walter also described. Engaged students also sought further information from reading, as Ruth and Jean mentioned.
Some supervisors also mentioned evidence of feedback-seeking, self-assessment, and wanting to improve by very engaged students. Nancy described a student “actively seeking feedback on how he was doing and what he could improve”. Oscar described a student who asked for “tips” about the areas she felt she was having trouble with, and then deliberately practiced and improved on those areas after feedback. Walter described a student who self-identified areas for improvement and deliberately challenged herself in those areas by her choice of topic for a