Validity in mixed-methods research

When qualitative and quantitative data are used together in mixed-methods research design, there is concern about the validity of merging (or connecting) data of different types (Cresswell and Plano Clark, 2011).

The very act of combining qualitative and quantitative approaches raises additional potential validity issues that extend well beyond the validity concerns that arise in the separate quantitative and qualitative methods procedures (Cresswell and Plano Clark, 2011, p.239).

Cresswell and Plano Clark (2011) make a distinction between data that are merged and connected, providing tables of the potential validity threats and strategies for minimising this for both types of data. Those judged relevant to my research design have been collated and personalised for my own research in Table 12.

136

Table 11: Three specific threats to validity and reliability of the quantitative measures used (adapted from Cresswell, 2009, p.164-165)

Type of threat Outline of the threat Application to my study

Instrumentation The instrument changes between pre- and post-test, which may affect the scores. The same instrument should be used in the same way

Almost all pre- and post-test measures were identical in this study, thus minimising this threat. However, the Pupil Attitudes to Self and School (PASS) questionnaire used (see Section 7.4.2) can be administered either manually or using a computer. Class A used a computer for the pre-test and a paper version of the PASS questionnaire (due to time constraints) for the post-test, which may have affected the validity of the findings. Testing Participants become familiar

with the outcome measure and remember responses for later testing. A long time period should occur

between administrations or different items used each time, where two versions of the same standardised test exist

There was a long time (i.e. several months) between the pre- and post-testing in my study, so earlier responses are unlikely to have been remembered, especially in the PASS

questionnaire, where there are fifty items in the test. In the second teacher interview, the participants were not reminded how they had responded to the same questions in the previous interview before the intervention started, including their ratings for how much they believe the class participates in decisions about their learning and how much they believe children should participate in decisions about their learning. Whilst this ensured that the teachers and children responded freshly to these questions each time, it could be argued that the rating scale is so subjective that their ratings would reflect more accurately the scale of any changes in their perceptions if they had been able to see what they had said before.

Interaction of selection and/or setting with treatment

Because of the narrow characterisations of

participants…the researcher cannot generalise to other groups of individuals. Claims should be restricted to groups of people with the same characteristics

The two schools and three classes and teachers all have many different characteristics. What unites them is: whole classes of Key Stage 2 children; teachers who are willing and interested in using AI; head teachers who are supportive of using AI to facilitate increased pupil

participation; Ofsted reports that suggest both schools require improvement; educational psychologist facilitation of AI; and late timing of AI (towards the latter half of the school year). Any findings that relate to all three schools may potentially be generalisable (or at least worth testing) in other schools that share some of these characteristics. Claims from the study cannot, however, be made directly about using AI with: smaller groups of children within a class; much younger or older children; teachers or leaders who are not receptive to the idea of using AI; schools that have been judged by Ofsted as outstanding, good or in special

measures; facilitation without an EP; earlier timing in the year etc. However, new research can/should be instigated in these very different groups and contexts.

137

Table 12: Strategies for minimising threats to the validity of merged or connected data in my study

Potential validity threats

General strategies for minimising the threat

Strategies for minimising the threat in my study

Making illogical comparisons of the two results of analyses

Find quotes that match the statistical results.

I used transcripts of teacher interviews and the children’s evaluation forms to match qualitative data with the results of quantitative analyses.

Giving more weight to one form of data than the other

Present both sets of results equally or explain why one provides a better understanding of the problem.

I used high quality reflexivity (see Section 6.5) to ensure that one set of data is not privileged over another according to my own beliefs and aspirations for the study.

Not relating the stages or projects in a

multiphase study to each other

Consider how a problem or theory might connect the stages or projects in an overarching way.

I ensured that data were analysed and compared across all three classes as well as within each case.

Using inappropriate sample sizes for the qualitative and quantitative data

Use a large sample size for quantitative data and a small sample size for qualitative data.

I used PASS questionnaires and simple rating scales for the large number of children involved, with semi-structured interviews reserved for the small number of adult participants.

Choosing inadequate participants for the follow up who cannot help explain significant results

Choose individuals for the follow up who participated in the quantitative first phase.

The teacher for each class commented on the quantitative results for their class. The children could not easily be consulted as they had left the class by the time the results were analysed. The timing of this study was a significant limitation in this respect.

Choosing weak quantitative results to follow up on

qualitatively

Choose the results to follow up that need further explanation.

The final teacher feedback meetings were focused on the quantitative results that suggest clear outcomes (either in support of or discrepant from the expected outcomes).

Including qualitative data in an intervention trial without a clear intent of its use

Specify how each form of qualitative data will be used in the study.

Teacher interviews were designed to explore any changes in beliefs about pupil participation and to allow the possibility of using direct quotes to illustrate key findings.

Not taking full advantage of the potential for “before” or “after” qualitative data findings for an intervention

Consider the reasons for using qualitative data in an intervention trial.

Pre and post teacher interviews allowed for self-assessment to be explored in terms of shifts within a theoretical model of pupil

participation and any changes in ratings to be discussed.

138 mixed methods results

in light of the advocacy or social science lens

the beginning of the study and advance a call for action based on the results.

heavily to my original research purposes of facilitating shared power between children and teachers.

6.5 Reflexivity

Reflexivity requires an awareness of the researcher’s contribution to the construction of meanings throughout the research process, and an acknowledgement of the impossibility of remaining ‘outside of’ one’s subject matter while conducting research (Willig, 2008, p.10).

Reflexivity recognises that researchers themselves, their beliefs, their values, their past history, their relationships with participants and their intentions are always part of their research and that this ‘reflexive’ perspective needs to be critically analysed throughout (Cresswell, 2009). Similarly, the same is true of every other person taking part as co- researcher. This is particularly important to acknowledge and explore within interpretative research but is even more significant in research that aims to be emancipatory (Usher, 1996).

Reflexivity suggests that researchers should acknowledge and disclose their own selves in the research…Highly reflexive researchers will be acutely aware of the ways in which their selectivity, perception, background and inductive processes and paradigms shape the research. They are research instruments’ (Cohen et al, 2007, p.171-172).

Researchers need to be self-consciously and critically aware of the effects they are having in a participative research process and how their attitudes, perceptions, opinions and feelings may be affecting other people involved (Cohen et al, 2007, p.310). Nind and Todd (2014) applaud the reflexivity of educational researchers who ‘address the messy detail and sticking points in

139

the reality and rhetoric of doing research that is intended to be participatory or emancipatory (p.2). Reflexivity may even open up new lines of action within the research (Bryant, 1996).

By showing that a statement is grounded in reflexive interpretative judgements, rather than in external facts, I make it possible to review other possible interpretative judgements concerning that statement, and thus to envisage modifying it (Winter, 1989, cited in Bryant, 1996, p.113).

At every stage of my research, I was aware of my influence as an EP and as a promoter of pupil participation. My status as the school’s EP in one school and as a literacy project co- ordinator in the other school is likely to have exerted an influence on participants’ perception of the freedom they had to express views different from my own. Additionally, it was not possible (or desirable) to hide my enthusiasm for AI or my passion for pupil participation. However, I have been equally passionate about my authentic desire for the teachers and children to feel able to be totally honest about their own beliefs, which may be very different from my own. From the very first introduction to the last interviews, I emphasised this total acceptance and respect for other people’s views. Indeed, after reading that the greatest effects of an AI are likely to be seen within organisations that do not share a participative culture (Bushe, 2010b), it became preferable to enlist teachers who did not share my views. I was therefore able to be completely genuine in my expressed desire for teachers to hold a variety of different beliefs at the start of the study.

In education, we need to be aware of reflexivity because even when we think our research is useful or even emancipatory we are still ‘objectifying’, still speaking for others, and education is full of people who speak for others in the name of doing good by them. Thus an awareness of reflexivity enables us to interrogate our own practice of research, in terms of how it can become part of dominant and oppressive discourses through a ‘reflexive’ acceptance either of the neutrality of research, of its ‘pragmatic’ usefulness or its ‘emancipatory’ potential, and in terms of how we contribute to such discourses despite our best intentions (Usher, 1996, p.49).

140

Willig (2008) and Usher (1996) make a distinction between personal reflexivity and epistemological reflexivity. Personal reflexivity means thinking about not only how the researcher’s own beliefs and interests shape the research but also how the research in turn changes or influences the thinking of the researcher. Epistemological reflexivity means reflecting on the assumptions the researcher has made about knowledge and the influence of this on the research questions, the design of the study and the data collected; asking questions such as:

How has the research question defined and limited what can be ‘found’? How has the design of the study and the method of analysis ‘constructed’ the data and its findings? How could the research question have been investigated differently? To what extent would this have given rise to a different understanding of the phenomenon under investigation? (Willig, 2008, p.10).

Reflexivity, in this way, strengthens the integrity of the research and is ‘key to the research act’ (Usher, 1996, p.42).

141