Validity and reliability

Validity and reliability in AR are complex issues (Denzin & Lincoln, 2005; Newton, 2006) that are connected to ethical questions stemming from the insider role of a researcher (Somekh, 1995). In AR, practitioners explore their own practices. This role may result in a narrow barrier between research and practice that may affect the analysis and interpretation of data. Although few writers have argued that AR is not qualitative (Greenwood & Levin, 2007), the mainly

The concept of validity is rooted in statistical scientific domains (Heikkinen et al., 2012) and may seem unclear to qualitative researchers. Richards (2006) argued that qualitative research is ‘soft’ because

its data sets have a plastic quality that yields to easy moulding. Far from making life easy for the researcher, this demands a particularly rigorous approach and constant vigilance in order to resist the seductive appeal of the superficial. At its worst, qualitative research does little more than state the obvious; at its best, it takes us close to the heart of things.(‘IATEFL Research Sig Newsletter’, para 1)

Such nature of qualitative research necessitates not only ‘adherence to correct procedures’ but also awareness of oneself as a researcher in terms of ‘how [he/she is] positioned in relation to the data and [his/her] analysis…and in respect of the craft skills [he/she is] continually honing’ (Richards, 2006, p.3). Other advocates of qualitative research, including Denzin and Lincoln, have argued to disregard the concept of validity in AR (as cited in Heikkinen et al., 2012, p. 6). Their stance is that AR’s validity is in its impact rather than in its statistics, which may not even exist.

A set of principles for validity and reliability in AR projects have been suggested by several writers. Regarding the validity of this study, the procedures suggested by Heikkinen et al. (2007; 2012) and Feldman (2007) were found useful. Heikkinen et al. (2012, p. 8) defined five principles of a quality AR, namely, historical

continuity, reflexivity, dialectics, workability, and evocativeness. Feldman (2007, p. 21) criticised these principles as ‘insufficient’ and argued to accept the claims that AR is mainly qualitative. Within this qualitative approach, validity is considered a construct that evaluates the quality of qualitative research studies, including AR. He suggested four new principles that would strengthen the validity of AR studies:

1. Clear and detailed descriptions of how and why data were collected

2. Clear and detailed descriptions of how narratives were constructed from the data

3. Seeking other ways to represent the same data and to use them to critique the views that one owns

4. Providing an explanation on why the actions led to the results, and an explanation or theory of why it works and that must be subjected to critique Richards (2006, para. 12) recommended Howe and Eisenhart’s (1990) list as a succinct summary of the essential qualities of sound qualitative research:

 Data collection and analysis techniques must be appropriate to and driven by the research question.

 These techniques must be applied competently.

 Studies must be assessed in the context of current knowledge, and aspects of the researcher’s subjectivity must be made explicit.

 An overall warrant for any claims must be established and will include consideration of disconfirming evidence and justification for the theoretical position.

 The researcher must be able to demonstrate an internal and external value. The former relates to research ethics, and the latter, to the relevance of the research to educational practice and its accessibility to actors in the relevant setting and to other researchers.

The validity of this study was achieved through the detailed accounts of how data were analysed and the continuous reflection on procedures and findings. The feedback forms, which were filled by the trainees and the latter’s journal entries were also sources of data upon which I reflected to improve further my practice. Reliability was achieved through the selection of methods that are compatible with the goals and limitations of each stage of the AR cycle. For example, survey, focus groups, and interviews were selected for the planning stage; journals, for the intervention; and follow-up and interviews, for the evaluation. Triangulation was also employed by cross-referencing the data from different methods.

Triangulation is a valuable strategy in terms of validity and reliability of studies (Johnson, 2007). The variety of data collection methods, which were employed in this study, such as surveys, focus groups, and journals, strengthened validity through triangulation. A set of principles or key performance indicators (KPI) were defined from the start. As for member check or peer review, the procedures were reviewed regularly by my supervisor and at times checked by another fellow teacher

trainer from the College of Education at Qatar University. Field notes, interview scripts, and a research journal were used to provide a detailed description of the study procedures. Finally, McTaggart (1997) noted that establishing credibility among and between participants is one way of validating AR. The establishment of credibility was initiated by providing teacher trainees with thorough information and by facilitating their role.

As regards the external validity of this study, previous studies have mentioned that validity is probably the weakest aspect of AR because of AR’s tendency towards small-scale projects that could not be generalised (Burns, 2005). To address such weak point, this study provided a detailed account of the procedures and session contents for accurate replication as seen in chapter 5 (e.g. Table 5.6, Appendix 5 and Appendix 6). The participation of several stakeholders in this study and the potential impact on their actions was the key factor of external validity. Greenwood and Levin (2005) noted that validity and reliability in AR are measured by the willingness of local stakeholders to act on the results of the AR study, thereby ‘risking their welfare on the ‘validity’ of their ideas and the degree to which the outcomes meet their expectations’ (p. 54). Trainee participants were asked to join ‘willingly’ this study and receive INSET on certain pedagogical issues related to ELT. Finally, construct validity was achieved through the selection of appropriate methods in the different stages of this study, employing both quantitative (e.g. survey) and qualitative (e.g. focus groups, interviews, and journals) methods. Greenwood and Levin (2007) argued that in AR, surveys, statistical analyses, interviews, focus groups, ethnographies, and life histories are all acceptable, ‘if the reason for deploying them has been agreed upon by the AR collaborators and if they are used in a way that does not oppress the participants’ (p. 5–6).

Reliability in qualitative research is synonymous with consistency. From these perspectives, qualitative research is considered reliable if the research findings can be replicated by another researcher (Lewis, 2009). Altrichter et al. argued that judging reliability in AR can only be carried out on a restricted basis as a result of the unique nature of any individual project (as cited in Turnock & Gibson, 2001, p. 471). Consequently, AR researchers have tended to focus on examining validity. Turnock and Gibson (2001) believed that AR reliability is achieved through a

detailed explanation of the researcher’s role and leaving the judgment to the readers.

In this study, I maintained a detailed documentation of all research activities in the form of emails, journals, field notes, and computer files. Transparency was sought by offering the participants access to their own results and sharing with them research updates. Among other tools used in this study were email, social media, and a website dedicated to this purpose (www.manasreh.com). Unfortunately, the domain was discontinued after the implementation stage, and it has since been registered to another person. However, the screenshots in Figure 4.2 below were taken during the implementation stage.