Validity and reliability

Although the terms of ‘validity’ and ‘reliability’ are usually associated with quantitative research principles, qualitative researchers also need to pay attention to such concerns in order to produce research outcomes that are trustworthy and rigorous (Dörnyei, 2007; Golafshani, 2003; Merriam, 2009; Morse, Barrett, Mayan, Olson & Spiers, 2002). Following Lincoln and Guba’s (1985) concept of ‘trustworthiness’ in qualitative research, some researchers use the criteria of ‘credibility, transferability, dependability and confirmability’ in evaluating the quality of qualitative studies (Table 4.9). However, although these terms are particularly developed for qualitative inquiry, they share the common underlying rationale with the terms ‘validity’ and ‘reliability.’ Stating that these two terms should be maintained in qualitative research, Morse et al. (2002) claims that ‘introducing parallel terminology and criteria marginalizes qualitative inquiry from

mainstream science and scientific legitimacy’ (p.16). That is, rather than developing alternative labels researchers should clarify what the terms ‘validity’ and ‘reliability’ particularly refer to in qualitative inquiry. As I also embrace a similar view, I prefer to discuss the quality of this study in terms of ‘validity’ and ‘reliability.’

Four aspects of the quality of ‘empirical social research’ that are frequently listed in research literature are construct validity, internal validity, external validity and reliability (Yin, 2009). As I embrace a constructivist orientation, I value multiple, subjective realities, and as Golafshani (2003) underlines, ‘to acquire valid and

reliable multiple and diverse realities, multiple methods of searching or gathering data are in order’ (p.604). Bearing in mind the fact that the present study adopted an interpretive and context-bound qualitative case study design, these four aspects were addressed accordingly throughout the research project.

Terms proposed for qualitative research in particular (Lincoln and Guba, 1985)

Internal validity Credibility External validity Transferability Reliability Dependability

Table 4.9. Terms for quality criteria in qualitative research

Construct validity refers to being able to establish appropriate ‘operational measures’ for the concepts under investigation (Yin, 2009, p.40). Internal validity is concerned with ‘the question of how research findings match reality’ (Merriam, 1998, p.201). As stated in the section about epistemological and ontological orientations, the present study, indeed, does not attempt to ‘discover the reality out there’ but to describe and explain peoples’ constructions of reality within its scope. In this sense, validity is concerned with the inferences drawn from the data rather than the data themselves (Creswell & Miller, 2000). In order to enhance internal validity, I have

applied some strategies during the data collection period such as using multiple methods/sources to check my interpretation of participants’ behaviours and of the emerging findings from my observations, as well as spending sufficient time in the research site. Johnson (1997) states that using multiple methods (e.g. observation, interview) provides the researcher with ‘method triangulation’ (p.288), and using multiple data sources (e.g. multiple observations, multiple interviews) provides ‘data triangulation’ (p.289). In the present study, I used both strategies.

External validity deals with the extent to which a study’s findings can be ‘generalised.’ It should be noted that ‘generalisability’ is ‘more than a matter of counting’ (Lazaraton, 1995, p.465) and it ‘plays a different role in qualitative research than it does in quantitative research’ (Dörnyei, 2007, p.59). Qualitative case studies, by their very nature, aims at understanding ‘the particular’ in depth, therefore the results cannot be presented as valid and accurate for every similar context and/or subject (Merriam, 1998, 2009; Yin. 2009). That is, the aim of qualitative research is not to discover ‘the universal reality’ observed by the many (Merriam, 1998) but to investigate realty(-ies) observed within the research context. Therefore, the present study addresses the concept of generalisability in terms of the philosophical underpinnings of qualitative case study design, which adopts ‘analytic’ rather than ‘statistical’ generalisation (see Cohen et al., 2011; Yin, 2009).

One of the misunderstandings about the qualitative case study is that ‘one cannot generalise on the basis of an individual case, therefore the case study cannot contribute to scientific development’ (see Flyvbjerg, 2006; 2011). However, Flyvbjerg (2011) claims that even though it is not ‘formally generalisable’,

knowledge may be transferable to similar contexts (p.305). Unlike quantitative research, which seeks statistical generalisation, qualitative research usually relies on analytical generalisation. Here, some findings and ideas from the study can be generalised in a theoretical sense (Dörnyei, 2007; Yin, 2009). In other words, a theory developed within a particular case context(s) may be generalised to some broader theory and can help to make sense of similar situations. As Blommaert and Jie (2010) claims, although data and findings are obtained through highly context- depended and interpretive means, such data ‘instantiate a case, and such a case belongs to a larger category of cases’ (p.13).

Another conceptualisation of generalisability in qualitative research, which was addressed explicitly in this study, is reader, or user, generalisability. Merriam (1998) states that if the case study investigator provides a rich and detailed description of the process, context and findings of his/her study, readers can compare those with their own contexts and consider whether it is possible to apply those findings to their situations. This detailed description of the research process along with researcher’s accounts for theoretical orientation and reflexivity also helps reader to see the consistency or reliability of the research. In the present study, presenting rich descriptions of contexts and procedures of data collection and analysis, I investigated the notion of authenticity in two main cases each of which had a pair of embedded units. Furthermore, both within-case and cross-case comparison were carried out during and after the data analysis process to enhance external validity.

In a traditional sense, reliability refers to the extent to which the procedures and findings of research can be repeated with the same outcomes (Merriam, 1998; Yin,

2009). Merriam (1998) states that as social studies deal with human behaviour and ‘human behaviour is never static’, the term reliability is somehow problematic in such studies. In fact, it can be claimed that achieving absolute reliability in qualitative studies is ‘not only fanciful but impossible’ in terms of replicating the

results (ibid., p.206). Therefore, what is important here is that the results and interpretations are consistent and dependable with the data collected and analysed (Merriam, 1998, 2009; Nunan, 1992). In other words, as Richards (2003) states, reliability is ‘a matter of being able to depend on getting the same reading if we follow the same procedures’ (p.285). In order to achieve this, thus making the

process more reliable and dependable, I provided adequate documentation and rich descriptions of the research process.

As for the reliability of research methods during data collection, I addressed this issue in at least three levels. First, the research methods (e.g. classroom observation, interview) had been carefully designed considering the rationale behind using these methods and the interconnected relationships between them. Furthermore, these methods/techniques were continuously revised and improved during both piloting and main data collection phases. Second, I aimed to improve myself as a researcher through practising and reflecting on my practices during both data collection and analysis periods. As Merriam (1998, 2009) states, the researcher is ‘the human instrument’ in qualitative case study design so s/he needs to train himself/herself to enhance reliability. Finally, a detailed account of procedures was recorded throughout the study.

In short, I paid attention to issues of validity and reliability in order to promote the quality of this study. Furthermore, the following strategies were used in this study (for detailed lists of strategies used to enhance validity and reliability in qualitative research, see Creswell & Miller, 2000; Johnson, 1997; Merriam, 2009):

Triangulation

Multiple research methods and sources were used for data collection (i.e. interviews, classroom observations, documents and research diary). The relationships between data obtained through these methods and emerging findings have been checked throughout the study.

Member checking

Tentative interpretations were sometimes shared with the participants from whom the data had been obtained. I believe that multiple (formal and informal) interview sessions also provided invaluable feedback from the participants about how accurately participants’ realities had been interpreted. Adequate

engagement in data collection

The main data collection was carried out in two phases over a period of time. The tentative numbers of interviews and classroom observations were set according to obtain enough data.

Researcher’s reflexivity

This involves awareness of and self-reflection on the potential biases and personal assumptions that may affect the research process. In order to raise my awareness, I kept a researcher’s diary during both data collection and analysis periods. I also combined it with the field notes whenever it was necessary or appropriate.

Audit trail

The procedures and ‘decision points’ in data collection and analysis were aimed to be recorded clearly. During this process, I sometimes consulted my supervisor as well.

Rich description

I aimed at providing a sufficiently detailed description of contexts and procedures in order to allow readers to determine whether findings and implications can be transferable in their own contexts.

Table 4.10. Strategies for validity and reliability