Ensuring methodological soundness in mixed method research

Arguably, the most important aspect of any empirical study is the extent to which the findings are sound and can be trusted. This is as important in qualitative as in quantitative research, and in order to ensure this within the current study, a number of factors were taken into consideration. However, certain challenges existed, particularly with regard to the TSQ and TWEQ, neither of which has established psychometric support (Hutchings et al., 2013). Whilst small sample sizes can impact reliability estimates of measurement tools (Charter, 2003), in this study both the TSQ and T-POT displayed acceptable reliability and internal consistency. Designed by the programme developers to examine participant workshop evaluation across a number of different and unrelated components, the TWEQ displayed poor internal consistency at almost all administration time points. Moreover, as TWEQ data were returned along with TSQ data (post-intervention), it is possible that teachers retrospectively completed the TWEQ questionnaires (in June 2009) rather than after each individual workshop, and consequently may have been subject to recall bias.

Rigour is the process by which high standards are met in qualitative research design (Montgomery, 2004). A number of criteria for rigorous qualitative research have been identified as important and these have guided the current study. They include credibility, transferability, dependability, and confirmability which, in effect, “...correspond with internal validity, external validity, reliability, and objectivity, respectively, in quantitative research” (Nastasi & Schensul, 2005, p. 185). The ways in which each of these was incorporated into the current study, are described below:

1. Credibility: Methodological triangulation is often used to examine convergent and divergent findings and was used in the current study to enhance credibility. Since a primary purpose of this research was to understand stakeholder experience and classroom outcomes, interview data on perceived outcomes were triangulated with both classroom observation and questionnaire data to provide credible findings. Respondent validation is another method by which credibility may be assured but it was not used in the present research due to its longitudinal nature and the possibility that early presentation of interview data to teachers could have impacted negatively upon future interviews and classroom observations. However, as an

alternative, an expert qualitative analyst conducted a secondary analysis of a small selection of baseline transcripts. A subsequent discussion of their findings with those of the researcher indicated a high level of agreement with regard to the identification of participants’ beliefs and experiences. Lastly, the use of verbatim quotes in the findings chapters (and in previous publications and presentations), facilitates credibility.

2. Transferability: Although it is not suggested that findings of qualitative studies can necessarily be generalised to the wider population (i.e., in the same way that generalisability may be a feature of many quantitative studies), the establishment of transferability was an important aspect of ensuring rigor within the current study. Transferability refers to the extent that findings may be applicable in other contexts. Here, this was enhanced by providing detailed information on the study context, for example, including an account of researcher-participant relationships and interview locations. This thick description is important in enabling others to determine the extent to which the findings reported here may be applicable to other situations and contexts. 3. Dependability: This third criterion refers to the extent to which the findings of

qualitative research may be considered reliable. From the initial stages of the study (i.e., when identifying and finalising the aims and objectives) through to the development, execution, analysis, and reporting of the research a detailed record of all activities was maintained by the researcher. All of these were important in ensuring dependability of the conclusions reached.

4. Confirmability: This final criterion is important in enabling others to determine the extent to which the findings have been influenced by researcher bias. Here, the development of audit trail was critical. This consisted of personal notes, process notes, and interview schedule development information. For example, detail as to how and why analytical decisions were made, is important in terms of replication of the study. Furthermore, this is important in helping to enhance researcher reflexivity (Watt, 2007); this is discussed in more detail in the following subsection.

The above were further complemented by the use of the ‘consolidated criteria for reporting qualitative research’ (COREQ) checklist (Tong, Sainsbury, & Craig, 2007) which contains 32-items across three major domains relating to researcher

characteristics, study design, and analysis/findings (see Appendix 18). This checklist was used throughout the analysis and write-up stage of the current study. An example of one COREQ item relates to the reporting of findings: “Were participant quotations presented to illustrate the themes / findings? Was each quotation identified?” The use of this checklist facilitated a comprehensive process of quality control throughout this research.

6.5.9.1 Reflexivity

A researcher’s own subjective biases and influences can unconsciously impact upon the research process; thus, reflexivity allows the researcher to identify and overcome subjective biases that may impact upon the analysis and presentation of findings. According to Charmaz (2006), no analysis is neutral and there is always the potential for contamination. Furthermore, Etherington (2006) noted that researchers have the potential to influence the very thing which they purport to examine. One potential (negative) consequence of failing to adhere to reflexive practice in research, is the risk of “obscuring the subjective”, and the presentation of findings “intentionally or otherwise, as objective” (Newton, Rothlingova, Gutteridge, LeMarchand, & Raphael, 2012, p. 880). Put simply, the way in which questions are posed may itself prompt particular cognitive processes and responses in participants.

For the above reasons, the researcher engaged in a process of reflection throughout the study (as recommended by Tracy, 2010). For example, the researcher had an appreciation of how classroom (mis)behaviour can affect teacher well-being. Furthermore, the researcher’s health psychology background and prior training in trauma counselling were helpful in facilitating an appropriate degree of empathy when teachers discussed the impact of chronic occupational stress on their psychological and physical health. It is possible, therefore, that empathy with teachers’ struggles and achievements may have encouraged participants to focus too much on this aspect rather than on other components of classroom experience. However, the researcher made a conscious effort to avoid superimposing her own expectations onto participants’ accounts. Furthermore, she attempted to remain mindful of her own potential biases throughout and, as previously discussed, the process of keeping an audit trail was invaluable in enabling this process of critical reflection.