2.2 Methodology overview relating to this thesis
2.2.3 Thistle Wood School case study (Chapter 4)
Thistle Wood School case study, described in Chapter 4, explored the research questions:
1. What are the teacher and student experiences of carrying out the Scottish CfE National 5 Assignment?
2. Which PISA competencies are developed and assessed in the Scottish CfE National 5 Assignment?
The study took a mixed methods approach and was an embedded, mainly qualitative case study. The mixed methods design was convergent, meaning that the two data types were collected together and analysed separately. The case study can be described as instrumental, which means it was used to contribute information to an overall research. The teacher and student experience was explored using qualitative data collection and analysis due to the interpretive rather than descriptive or comparative nature of the research question. Exploration of the PISA sub- competencies used was descriptive and comparative and so was explored using quantitative data collection and analysis.
Nineteen secondary school science teachers participated in the main study and provided secondary documentation relating to 402 of their students (a pilot was carried out the previous year). Data was collected using a mixed methods questionnaire which included a series of open- response questions that aimed to gather qualitative data, followed by closed response questions, with an optional open response, that aimed to gather both quantitative and qualitative data. Examples of completed questionnaires can be found in Appendix A. Secondary documentation included: anonymised student work and student evaluations, also in the form of a mixed methods questionnaire; departmental lesson plans and schemes; teacher quality assurance materials, such as minutes of meetings and lesson evaluations; and researcher field notes. All data was analysed using thematic analysis and a detailed method is described in the following paragraphs (ii).
(i) Thematic analysis used in this thesis
Exploration of the teacher and student experience required the researcher to interpret rich qualitative data and so thematic analysis was chosen. Thematic analysis was carried out on the qualitative data gathered: open-response sections of teacher and student questionnaires, teacher interviews and secondary documentation. The method of thematic analysis used in this study was initially based on the method described by Braun & Clarke (2006) but was refined as a result of a pilot study. The final method of thematic analysis devised for use in the studies described in this thesis is outlined in Figure 2-5 and the following paragraphs. This method of thematic analysis was used in each of the case studies (Thistle Wood School, Clover Field School, Daisy Park School and PST case study).
Figure 2-5: Thematic analysis method used in this thesis
Each type of data (e.g. questionnaires, interviews, documents) separately underwent a systematic process of analysis as shown in (Figure 2-5) before themes were combined to an overall account of the “case”.
Firstly, the researcher read and transcribed (if necessary) the qualitative data and noted initial ideas and thoughts. This was known as the initial ideas stage. At this stage the “codebook” began to form with overall headings or themes and broad statements of what may be included.
Next, in the initial themes stage, the researcher formed themes from the initial ideas and began a process of systematic coding of data extracts (quotes, pictures, snapshots, video and audio extracts) into each theme. This coding process continued until all data from the data source (e.g. questionnaire) had been coded. It was possible at this stage to have a theme such as “other” or “miscellaneous” to allow all data to be coded. At this stage the codebook was detailed and inclusive but there may have been overlapping sub-themes or sub-themes that were too small (i.e. too few references or participants referring to it) or too large (i.e. contained multiple topics and would be better as separate sub-themes). At the end of this stage member-checking occurred, where the initial themes and sub-themes were discussed with the participants. The teachers were asked whether the themes represented the phenomena, in this case the experience of the National 5 Assignment, as they perceived it and any ambiguous answers or industry specific terms were
checked for meaning. These conversations were used to inform the identification of the final themes.
Thirdly, the researcher reviewed the themes and sub-themes by checking all coded extracts fitted within the theme they were allocated. The “other”/ “miscellaneous” theme was checked and extracts were either coded into an appropriate theme or discarded if not deemed relevant. The themes were given clear descriptions and names, which were written up in a codebook. References that were found in more than one sub-theme of the same theme were placed into the most appropriate sub-theme and this decision this was noted in the code-book. By the end of this stage the codebook described clearly what should and should not be included in each theme and sub-theme, and the references/extracts within these matched the codebook.
The final stage of analysis was when all data sources were analysed together. The themes and sub-themes were combined for all data types. All references/extracts were re-checked against the allocated theme/sub-theme and the codebook to ensure consistent coding across all the data types. Appendix D shows the full, final codebook for Thistle Wood School case study.
There were three considerations for the final themes and sub-themes, adapted from content analysis procedures, that aimed to make the process more manageable and systematic:
1. Mutual exclusivity: The sub-themes within a theme are mutually exclusive and one unit of coding cannot be coded into multiple sub-themes.
2. Exhaustiveness: All data is assigned to at least one sub-theme. This applies to initial themes only as extracts from the “other”/ “miscellaneous” theme may be disregarded as irrelevant or not representative of the overall account before finalisation of the themes.
3. Saturation: All sub-themes have at least one relevant extract, no theme or sub-theme can be empty. In larger case studies (e.g. Thistle Wood School case study) the sub- theme should be present in more than one of the participants’ accounts.
(adapted from Schreier, 2012)
Triangulation was used to increase validity of findings by comparing findings between data sources, i.e. surveys and interviews with teachers, and between participants, i.e. students and teachers, teachers of different subjects, teachers of different roles. Validity was also increased as themes were checked with the participants, member-checked, to ensure they accurately represented their experience before being finalised.
Reliability was considered and increased through a systematic and consistent approach to coding. This was achieved through rigorous procedures as shown in Figure 2-5, including the development of a detailed “codebook” that outlined what should be included in each theme and sub-theme.
Inter-rater reliability was not used in this study as it was not deemed as appropriate given that the themes were interpretative. Instead themes were checked with the participants themselves. However, at appropriate stages in the analysis the working themes were subject to consultation with academic supervisors.
Data was also analysed quantitatively in this study. The percentage of teachers and students who responded “yes” to questions 7, 8 and 9 and who reported using each sub-competency in the N5 Assignment (part two of the questionnaire) was calculated (Appendix A). This information was displayed graphically.
This study underwent “expedited” ethical review. The participants in the study were teachers being questioned in their professional capacity about their professional activities and as such this aspect was considered low risk. Data collection methods were also considered low-risk: questionnaires and interviews with adults and document analysis. Surveys of student views of educational experiences are carried out as standard by teachers as part of quality assurance procedures in the participating school. The information collected was not of a sensitive nature but related to students under the age of 18, so this aspect was considered moderate risk. Note that all schools and participants have been given pseudonyms to ensure anonymity.