This chapter begins an account of action research that will continue over the next two
chapters. It describes participatory action research performed with colleagues, as part of a
shared transformational journey. The interventions were agreed based on evidence
presented at the end of each cycle of action research. It was imperative that this data was
accurate, reliable and shared in a clear format to enable the steering group to agree informed
decisions on the best way forward. This chapter outlines the process of data collection and
local dissemination.
Action research is not just about achieving change, or this would simply be a management exercise. To merit the title ‘research’, this study must be more than a reflective account of change, rather it must represent ‘a systematic investigation, a way of knowing that lays bare its methods for all to see’ (Parahoo, 2013:10). The decision to include a short chapter on the story of data collection, analysis and local dissemination was based upon a recognition and agreement with Morton Cooper (2000:81) that one of the most challenging parts of action research is the process of evaluating and transforming the large volumes of data into a readable and effective report. The chapter is used to establish the cultural validity of the data by explaining how it was collected, interpreted and utilised as part of transforming the learning ecology8.
131 The chapter will share the systematic processes used to critically examine and interpret data, generating the evidence which informed subsequent research cycles. In a disciplined manner, the issues for investigation were agreed, questions defined and the collected raw data analysed by searching for key themes or trends. This was disseminated to the Programme Committee at the end of each six-month research cycle, providing a collective forum to decide on the significance of data, agree what (if anything) had been learnt, if any actions or changes were now warranted and what we needed to learn more about. This critically evaluative process included ongoing questioning of whether the research tools were fulfilling the data collection functions for which they were designed and if they were posing the right questions. The quality and focus of this data was an acid test of the appropriateness of the action research methodology, and the ability of the research instruments to inform and evaluate the changes made to the curriculum.
What Counted as Data?
Data refers to all the items of information gathered during the research project (McAteer, 2013). Data sets relating to each cohort and theme were collated by combining the information from the focus groups, questionnaires and classroom observations. The quality of this data constitutes the ‘evidence’, which forms the basis for any claims made by this research.
As Gray (2014) explains, action research is a modern approach to research, concerned with not only analysing practice, but also trying to change it. In practice, this involved identifying and addressing problems with the other participants, being involved as a change agent, rather than a detached scientist. This immersion resulted in a qualitative stance, working closely with the MCCC team to learn and improve the programme together. To provide
132 assurance of the integrity of the data, it is necessary to provide a clear and transparent record of how this information was collected. Gray (2014) advises that data gathering should be systematic, with a permanent record of what took place, and recommends using a variety of methods (as in this study) to allow data triangulation.
Multiple Sources of Data
This data was derived from multiple sources including students, PBEs, lead nurses, classroom observations and programme key performance indicators. Combining evidence from these varied sources provided convergence of data, allowing us to benefit from hearing different perspectives, whilst retaining a level of objectivity by observing the reality of classroom teaching and learning practice. The convergence of data from both multiple sources and research methods provided a holistic view, engendering confidence that the data was authentic and a trustworthy evidence base to inform decision making.
Students formed the main source of data within this study with 237 contributing their time and energies to providing this data set. They completed 278 questionnaires, with a further 62 students participating in 12 focus groups (28% of the student population). Classroom observation of each of the 12 study days was performed over the course of the first year of the study and repeated in the second year (n=24).
Key performance indicators
• Student academic performance in summative assignments (first attempt)
133 A personal diary
A personal diary is a key feature in action research (Elliot, 1991; Whitehead, 2002; McNiff, 2013). It provided a detailed narrative of events, a chronological record including reflections and interpretations of events. This was cathartic in capturing feelings, and proved important in pulling strands of information together; this reflexivity (see Parahoo, 2013; McAteer, 2013) generated personal insights relating to my role, control, democracy and the need to retain focus on ways of empowering rather than oppressing participants.
Data Collection
The data collection began with focus group interviews with the PBEs on August 22nd 2013 and finished when I interviewed them again at the end of the study on 29th September 2015. A research plan provided a detailed map with valuable milestones. Consent forms were collected from all participants, numbered and stored together. The focus group recordings were transferred digitally to my computer for future analysis and transcription. All the data for each cohort and each cycle was collated together and stored securely. The data was processed in a timely manner, driven by the need to evaluate our interventions and feedback to the steering and working groups.
Recording and Transcription of Focus Group Recordings
The focus groups were audio recorded; the recordings were stored securely before being erased from the recording device to maintain confidentiality. Video recording was not used as it was considered too intrusive as it may have inhibited participation and contributions. A quiet environment was achieved for all the interviews and the quality of the audio recording achieved was excellent. These audio recordings were transcribed verbatim by the principal researcher using Dragon dictation software, which was accurate and easy to use. The
134 transcriptions produced using this software required close observation and regular correction. The recordings were therefore listened to alongside the transcripts to confirm missing or inaccurate data (Bryman and Bell, 2015).
This process involved some trial and error. During the first focus group, participants were asked to state their first names at the start of the interviews to allow identification during transcription. In practice, this proved inadequate to allow clear identification of participants and subsequent groups were asked to speak a full sentence, which made it easier to detect different accents and tones. This was not highlighted in the literature, but from a practical perspective it was essential that I could identify participants, and this can be problematic in larger focus groups. The objective was to produce an accurate representation of both individual contributions and group conversations. Redmond and Curtis (2009) explain this means considering how participants spoke, keeping to the language they used, the intensity of their voice and feelings about the topic.
The transcriptions were coded with each student allocated a different colour, assisting analysis of everyone’s contributions at a glance, and highlighting the interventions of the moderator, providing increased visibility on my role as a moderator. The intonation of the person speaking was illustrated using underlining to indicate emphasis and brackets to provide further description, such as group agreement, a pause or laughter.
Questionnaire Response Rates
A common pitfall of using questionnaires is the potential for low response rates (Dillman, 2007; Bowling, 2014; Bryman and Bell, 2015). Response rates are important because high response rates increase the validity of results by ensuring that the data collected represents the populations’ views (Cormack, 2015). Whilst acknowledging that these risks are
135 particularly associated with postal questionnaires, achieving high response rates was dependent upon the motivation and good will of the PBEs to distribute them and the students to complete them. A response rate of 50-60% is recognised as barely acceptable, and over 75% rates as good (Bowling, 2014) and over 85% excellent (Bryman and Bell, 2015). A summary of the response rates is presented in table 3 below.
Table 3. Summary of questionnaire responses by student cohort
Note* that as the CCP is a rolling programme with intakes every six months it was not possible to perform pre- and post-course questionnaires with each cohort. As the study commenced in September 2013, there were no pre-course questionnaires for the September and February 2013 cohorts; and because the study was completed in September 2015, there are no post-course questionnaires for the February 2015 cohort as they would not complete the programme until six months after the study was completed.
A good response rate was achieved by using the following methods: an inclusive approach involving all the students who were not participating in the focus groups; the use of cover letters sent to all students when they enrolled on the programme; and administering the
Student cohorts Pre -course response
rates
Post course response rates September 2012 N/A* 54% February 2013 N/A* 82% September 2013 100% 79% February 2014 100% 100% September 2014 100% 100% February 2015 100% N/A*
136 questionnaires by hand, resulting in a 100% response rate for pre-course students. An initial poor post-course response rate of 54% was caused by some educators agreeing and then ‘forgetting’ to distribute questionnaires, which may have reflected passive resistance or just forgetfulness. The remaining response rates were very good to excellent, reaching 100% by the February 2014 cohort.
Addressing Criticisms of Self-completion Questionnaires
There are several potential disadvantages associated with using self-completion questionnaires including a loss of control, which can result in non- or partial completion; or incorrect interpretation of the questions by participants, which can in turn damage the reliability and validity of the data. Puchta and Potter (2004:48) felt that questionnaires were less engaging than focus groups, constraining people’s responses and depriving them of the opportunity to fully express their views. This concern is valid, as the questionnaires did not provide the opportunity for follow-up questions to probe and explore the meaning of answers. It is accepted that there existed the potential for researcher bias either in the design of the questionnaire or during the transcription and interpretation of the data (Bowling, 2014). To minimise this and assist transparency, anonymised copies of the questionnaires were shared with the Curriculum Working Group for inspection before and after completion.
Key Performance Indicators
The quality indicators of student performance were derived from the cohort unit reports, which include details of unit and programme completion rates, pass at first attempt and attrition rates. These were presented to the Exam Board and Programme Committee every
137 six months and formed part of the Continuous Improvement and Programme Development plans. This provided a shared verifiable data source.
Data Analysis methods
Content and thematic analysis methods were used to analyse and converge the data from the classroom observations, focus groups and questionnaires into meaningful findings (see Boyatzis, 1998; Massey, 2011). Thematic analysis is frequently used in qualitative research, to identify, analyze, and report the patterns (themes) found within a data set (Braun & Clarke, 2006).This explorative and inductive approach to data analysis suited our study because it helped us to learn what the participants felt about key issues and generate data, such as voice excerpts that provided an accurate and authentic
representation of participant views to increase the trustworthiness of the data (see Elo and Kyngas, 2008). As the study progressed, I increasingly utilised content analysis methods, because they allowed me to quantify the large volume of data into meaningful and
representative statistics relating the participants responses to the issues surveyed (Vaismoradi, et al., 2013, Treadwell, 2014).
Content analysis was used to objectively determine the frequency of categories such as the number of times students used terms such as ‘evidence base’ and ‘confidence’; or to quantify the strength of feeling on some issues such as levels of ‘student motivation’ (see (Dixon-Woods, 2005; Elo and Kyngas, 2008). Content analysis techniques were valuable because the counting of participant responses in different categories generated descriptive statistics, which provided visual representation of facts across the study. It supported the analysis of data derived from Likert scales, such as median and range to interrogate the data more rigorously; generating greater insight into issues such as students’ perception of
138 their clinical and academic learning gain as a result of attending the CCP. This blended use of these two methods worked effectively because they share many similarities as illustrated in the table below (see Vaismoradi, et al., 2013). This approach led to the generation of educational themes that were important to meeting the aim of the study, such as ensuring critical care nurse education focuses of the care of patients and their families; evaluating the impact formative assessments have on student learning gain; and eventually to the development of broader concepts, such as that of transforming a learning ecology.
Data analysis
The data was collected, analysed and interpreted in synchrony with the action research cyclical process informing the subsequent interventions to the programme made by the Curriculum Working Group. A large volume of data was obtained from the 20 focus group recordings, 10 sets of cohort questionnaires and 24 classroom observations. A five-step method of data analysis was used, following the conventions of thematic and content analysis outlined by Braun and Clarke (2008; 2018) and Elo and Kyngas (2008), which are presented below.
Thematic analysis (Braun and Clarke, 2008) Content analysis (Elo and Kyngas, 2008)
Familiarisation with the data. Transcribing and
reading the data, noting down initial ideas.
Preparation. Obtaining a sense of the
whole, selecting the units for analysis. Being immersed in the data.
Generating initial codes. Coding interesting features
of the data systematically across the entire data set.
Searching for themes. Collating codes into potential
themes. Ongoing process of reviewing these themes, potentially to grouping into larger themes.
Organising & opening coding, creating
initial categories, formulating general descriptions, sub categories relating to the research topic
Reporting the themes and relating this back to the
research question
Reporting the analysing process and
results through models, categories and a story line
139 The data was divided into meaningful units for analysis; a process that is used in content analysis. Student focus group transcriptions and questionnaire responses from each cohort were grouped together to provide an overall perspective of the views of each cohort. Other data sets included the classroom observations or focus groups with all the educators. This identification of data sets was useful because it allowed student cohort or educator responses to be viewed as a whole (see Graneheim and Lundman, 2004).
Step 2. Familiarisation with the data
The focus groups produced a large volume of transcribed narrative, a 'thicket of prose’ (Bryman and Bell, 2015:424). Data was generated from the focus group recordings by repeated listening, alongside analysis of the printed transcriptions. This immersion enabled me to sort and categorise (code) the data, ‘listening’ to participant voices, leading to the identification and greater understanding of prominent themes (see Dixon-Woods, 2005; Braun and Clarke, 2006).
Step 3. Organise and generate initial codes (categories)
The process involved openly coding the data by making notes on the transcriptions and questionnaires, underlining key words or phrases such as ‘evidence-based practice’ or ‘improve patient care’. Data analysis was an inductive process, with categories generated from the presence of common terms or descriptors recorded by the participants, such as decision making, or broader themes relating to the basic nature of the acute care unit content.
140 As the study progressed, the initial categories and themes were reviewed to make sense of the data, and some terms or phrases were merged into single groups. An example is when participants used similar words relating to learning the research underpinning care, this was coded in the ‘evidence based care’ category (see Polit and Beck, 2012, Braun and Clarke, 2018). The occurrence of these categories and responses to questions was translated into statistics using content analysis. This produced descriptive statistics in the form of percentages derived from the closed questions and Likert scales used in the questionnaires and during focus groups. This process was based on the guidance offered by Bailey et al. (2014), with mean, median and range used when analysing Likert scale (1‒5) responses. The range was particularly useful in conveying the divergence of opinion on issues such as fairness of the CCP writing assessment or assessing participants’ level of perceived clinical confidence at the end of the programme, establishing these issues as significant themes.
Step 5. Reporting phase
In the reporting phase, these descriptive statistics were presented alongside excerpts, to construct statistical evidence, alongside the authentic voices of participants to the working group and Programme Committee. The convergence and subsequent interpretation of this information allowed us to identify key themes, analyse the meaning of this data in relation to informing the action research and generating new insights.
Dissemination
The data was presented to the key stakeholders on a cyclical basis in a consistent, anonymised and accurate manner, so that it could be easily understood. Validation of the data was examined through concurrent analysis and discussion within the Curriculum Working Group and Programme Committee. This was essential: the evidence informed
141 decision-making underpinning interventions that would impact not only on the quality of critical care education in Greater Manchester, but also, by implication, critical care provision. This placed a large burden of responsibility on me to ensure the evidence was balanced, verifiable, authentic and transparent. All were essential to maintain the integrity and credibility of the data over the two years of the study, providing the confidence for us to make changes and learn together as a team.
Summaries of the evidence from each research cycle were used to share the large volumes of data in a timely manner. This provided an effective solution to the problem reported by Morton Copper (2000) of transforming the volume and variety of data produced by action research into a readable and understandable report. This enabled myself and the MCCC team to maintain a collective overview. This effectively formed a shared action plan, providing the steering group with an analytical tool for framing the issues, keeping us abreast of, the evidence, before agreeing actions and identifying areas for further investigation. This helped maintain a shared focus when deciding the significance of information, or prioritising issues that required further exploration or intervention. The retention of an overview of the action research process was challenging, and different formats of the summaries of evidence were developed to assist this process; this was useful in capturing and