Research questions
At the outset of this research the central research question was:
What is the value and impact of formative assessment and formative feedback to the learning experience of students?
At that time, this primary research question was guided by the following specific questions within the pilot study:
What worth do learners and educators place on formative assessment and formative feedback within and across the learning spectrum?
What role does formative assessment and formative feedback play in the “teaching” experience of the educator?
Is there difference or similarity between the worth placed on formative assessment and formative feedback by learners and educators?
Is there difference or similarity between practice-based educators and university-based educators as to the role and merit of formative assessment and formative feedback? How is formative assessment and formative feedback used to support the summative assessment process across the spectrum of education?
What are the key features of formative assessment and formative feedback across education and how can these be implemented more widely?
Ethical approval
Ethical approval to undertake the pilot study was sought by the School [of Human and Health Sciences]’s Research and Ethics Panel (SREP) at the University of Huddersfield, as that is the University with which I am registered for this award. Ethical approval was gained on 24th April 2009.
Process of the pilot study
The pilot study involved student participants that were in their final year (during 2008-9) of undergraduate physiotherapy and/or podiatry students, and their educators. These two student cohorts were chosen as a convenience sample because both cohorts were relatively
154
large and readily accessible (Cohen et al., 2007) to me. In addition, there was a clinical facility within the university in which some of the students undertook authentic clinical experiences and this offered me a vehicle by which to pilot the participant observation of my methodology without the need for NHS ethical approval (Integrated Research
Application System (IRAS), 2011). The educator participants were those educators who supported the student participants in university-based and/or practice-based learning environments.
Data collection was a threefold process involving an anonymous electronic questionnaire, participant observation and a student focus group.
Electronic Questionnaire
All student participants (n=80) were invited to return an anonymous electronic
questionnaire via Blackboard, the university’s virtual learning platform at that time. The students were sent an email by me explaining the purpose of the anonymous electronic questionnaire and detailing what they were being asked to do. This email also explained that by returning the questionnaire, informed consent was assumed. The electronic
questionnaire data were analysed using Statistical Package for the Social Sciences (SPSS) and content analysis (Bryman, 2004).
Participant observation
Year 2 academic summative profiles of all the student participants were accessed and, based on these profiles, three student participants were purposively selected by an independent person. These students were purposively selected so that one student had average (modal) module marks in the 40% band or less, one student had average (modal) module marks in the 50-60% bands and one student had average (modal) module marks in the 70% band or above. The three selected student participants were invited by me, via an explanatory email and an information sheet, to be observed and video and/or audio
recorded in at least one authentic university-based or practice-based learning setting during which they were to receive feedback. At the time of undertaking the pilot, I perceived it essential that participant observation was tested in at least one university-based and one practice-based setting in order that any authentic spatial variations might be tested. The educator participant from whom the student participant was to receive feedback was also emailed directly by me and provided with an information sheet inviting them to be observed and video and/or audio recorded. Informed written consent was gained from all observed participants. The video and/or audio participant observation recordings were transcribed by
155
an independent person and the data analysed using a content analysis approach (Bryman, 2004).
Focus Group
The three observed students were invited (by email, with accompanying information) to attend a focus group which was audio recorded. A focus group schedule was devised and informed written consent was gained from all participants. The focus group had three purposes. Firstly, it allowed the opportunity to ask whether the question set within the electronic questionnaire was sound (i.e. logical, without ambiguity, complete). Secondly, it provided me with the opportunity to test the methodology of organising and running a focus group and managing the subsequent data. Finally, it gave me the opportunity to discuss with the three observed students their perspective of the observation. The focus group recordings were transcribed by an independent person and the data analysed using a content analysis approach.
Findings and methodological outcomes of the pilot study
The pilot study highlighted key aspects of the methodology that needed further
consideration. Firstly, the response rate to the electronic questionnaire was very low at 32.5% (n=26). During the focus group, the student participants suggested that if they had known who the researcher was and had met them prior to receiving the questionnaire they would have been more likely to complete it. This allowed me to consider alternative
methods of engaging with the students for main study. Secondly, the questionnaire
illuminated that 92.3% of electronic questionnaire respondents (n=24) viewed themselves to be solely or equally strongest in clinical practice (as opposed to academic work alone). This suggested that it was imperative for me to include feedback experiences within authentic practice-based learning environments during the data collection process,
particularly given that these environments are the crux of what makes healthcare student education different to that of other university undergraduate students.
Results
There were core themes that emerged from the content analysis. A discussion of each is below.
Same Difference?
Students view formative assessment and formative feedback synonymously, irrespective of whether the [formative] feedback is associated with formative or summative assessment methods. Students do not associate “formative” with “learning” and “summative” with
156
“verification”. Students are instead more likely to simply reflect that “formative” does not contribute towards a final grade/award classification:
Formative assessment is a method of testing students on topics that they have learnt so far that do not count towards final marks. (Student)
Analysis: there appears to be an unclear and/or artificial and/or unnecessary differential between formative assessment and formative feedback. Also, students fail to recognise implicit learning opportunities.
Implications: feedback – whether it is associated with summative or formative activity – should be at the core of a learning opportunity and this opportunity needs to be made explicit to students.
Message Understood?
Students verbally (e.g. “Yeah”/”Hmm”/”Yes”/”OK”) and non-verbally (e.g. nodding) suggest to tutors that they understand the feedback being given but then explicitly demonstrate that they have misinterpreted what is being said and/or remain unclear by the subsequent
questions they ask:
… ethics section is superficial … (Educator) It needs to be a lot longer … (Student)
Analysis: whilst the tutor is implying that the work needs to be different, the student is interpreting the message as it needing [only] to be longer.
Implications: if there is no opportunity to ask questions (e.g. if the feedback offered is in the written form alone), messages may be “lost in translation” from tutor to student.
Students then, whilst they believe to have understood the message correctly, actually retain an [ongoing and] incorrect understanding of the message. Feedback needs to be overt and explicit.
What is our Ethos?
Students and tutors both demonstrate an explicit assessment-driven, as opposed to learning-driven culture. Students clearly want to simply pass assessment tasks and the literature supports this (e.g. Ecclestone and Pryor (2003), and Davies and Ecclestone (2008)). Tutors are also [subconsciously] promoting an assessment-driven culture: It’s going to be capped at 40[%] regardless … (Educator)
157
Analysis: the tutor is prioritising the summative mark likely to be achieved over the ongoing learning opportunity offered.
Implications: students and tutors are at risk of actively nurturing (implicitly or explicitly) an assessment-driven culture over a learning-driven culture, at a detriment to the latter.
What Rules?
Students receive mixed messages from tutors in relation to what is expected of them and experience varied practices used by tutors to assess and provide feedback:
Again it’s not me that’s marked it, I’d have just … (Educator)
Analysis: the tutor is outlining inconsistencies between markers by suggesting that s/he would have done something different had s/he marked the work.
Implications: if the expectations and practices of assessors are different, students will find it impossible to understand the “rules of the [assessment] game” as the “rules” are likely to be constantly changing. This is unfair to the students.
Conclusions and the Next Steps
The findings of the pilot study led to the initial central research question being reviewed and honed to:
What is the perception of the value and impact of feedback on ongoing pre-registration healthcare student learning?
The additional research questions were reviewed to: How do healthcare students use feedback?
Do healthcare students understand what is intended by feedback? What do educators intend students to "do" as a result of feedback?
Does previous academic experience or level of attainment influence the impact of feedback? What feedback experiences and behaviours influence ongoing learning?
Is there a need for quality assurance processes for feedback and learning in practice-based and academic settings?
Given that the primary research question had been honed to understand perception of experience, it was felt that the questionnaire served very limited purpose towards answering this question. Rich descriptive data was wanted to understand the lived
158
useful a questionnaire would be towards this (Bryman, 2004) and this methodology was removed from the main study.
Key findings:
Formative assessment and formative feedback activities do not appear to have a consistent intended effect on the student learning experience.
The current “formative” focus of the research needs to be reviewed whilst the “feedback” element needs to be explicitly strengthened in the main study.
“Modes” of feedback need to be explored more robustly within the ongoing research such that the need for dialogue between the student and the tutor in order for the student to verify understanding can be investigated.
The ongoing research needs to fully explore the overt and covert “cultures” and practices associated with assessment and feedback from both a student and a tutor perspective.
159