Data validity and trustworthiness

Validity and trustworthiness of data and findings is a requirement of any type of educational research. Whether carried out within a positivistic or naturalistic tradition, or based on qualitative and/or quantitative sources of data, validity “is the term used to claim that research results have precisely addressed research questions” (Somekh & Lewin, 2005, p. 349).

Regarding trustworthiness, Lincoln and Guba (1985) indicate that this term is related to the issue of a piece of research being worthy of attention. In other words, research that is invalid and worthless is of low quality (Cohen et al., 2007). But then, what constitutes validity and trustworthiness?

Patton (2002) argues that it all depends on criteria. This is, research can be of quality or not depending on the ontological, epistemological, and methodological approaches adopted during the inquiry, as well as on the

approaches and criteria taken to judge the quality of such research.

Different research paradigms have different criteria to determine quality and validity of research.

For example, the concept of reliability, which involves the ‘possibility of replication’, is a common criterion within a positivistic research tradition.

But it cannot be accounted for in those terms within a naturalistic perspective, as such inquiry approach asserts that each real life context that can be studied is unique, thus, not replicable (Cohen et al., 2011). In qualitative research, which occurs within the natural setting, reliability can be regarded as a degree of accuracy and comprehensiveness of coverage of recorded data (Bogdan & Biklen, 1992). Cohen, Manion and Morrison (2007) further argue that reliability in qualitative research involves fidelity to the real life inquiry context, authenticity, honesty, depth of response and meaningfulness to participants. Moreover, Maxwell (1992) suggests that

‘understanding’ is a more appropriate term than validity, in qualitative research.

Lincoln and Guba (1985) describe four criteria of the ‘traditional’ positivistic paradigm that account for validity and reliability: internal validity, external validity, reliability and objectivity. Although Cohen et al. (2007) indicate that internal and external validity can be addressed both within quantitative and qualitative methods, Maxwell (1992) argues for five kinds of validity in qualitative research that explores his notion of ‘understanding’: descriptive validity, interpretive validity, theoretical validity, generalizability, and evaluative validity. Other scholars in the field of naturalistic inquiry describe different criteria for judging quality of inquiry. For example, Patton (2002) advocates for trustworthiness, authenticity, triangulation, reflexivity, particularity, and enhanced understanding, among others.

In the case of the present research, as a naturalistic tradition has been chosen as the inquiry paradigm, the four concepts defining trustworthiness in naturalistic paradigm indicated by Guba and Lincoln (1989) were adopted here as the criteria of validity. These four concepts of

trustworthiness are: credibility, dependability, confirmability, and transferability. Below follows a brief review of these concepts of trustworthiness, with an indication of its application and/or implication in the present study.

Qualitative research relies on description, interpretation, and explanation.

In this sense, credibility of inquiry is related to the issue of description, interpretation and explanation being credible (Janesick, 2000). Lincoln and Guba (1985) propose five techniques to increase the credibility of naturalistic inquiry (p. 301):

• Activities that make it more likely that credible findings and interpretations will be produced: prolonged engagement, persistent observation, and triangulation.

• An activity that provides an external check on the inquiry process (peer debriefing).

• An activity aimed at refining working hypotheses as more and more information becomes available (negative case analysis).

• An activity that makes possible checking preliminary findings and interpretations against archived ‘raw data’ (referential adequacy).

• An activity providing for the direct test of findings and interpretations with the human sources from which they have come – the constructors of the multiple realities being studied (member checking).

As discussed earlier in relation to the research design of this study, and as further discussed in Chapter 5 in relation to the intervention of this study, activities related to prolonged engagement (familiarity with inquiry setting) and persistent observation (identification of key elements of the study) were an integral part of this inquiry. Also included were activities and instances related to peer debriefing, referential adequacy and member checking. Regarding negative case analysis, where the object is to continuously “refine a hypothesis until it accounts for all known cases without exception” (Lincoln & Guba, 1985, p. 309), such an analytical and

interpretive process was also considered in this inquiry, and highlighted throughout the findings chapters (Chapters 6 to 8).

Finally, triangulation is a powerful analytical and validity tool in qualitative research consisting of the cross-validation of data obtained from different sources and perspectives (Somekh & Lewin, 2005). During the 1970s, Denzin (1978) identified four basic types of triangulation: data triangulation; investigator triangulation; theory triangulation; and methodological triangulation. Apart from the investigator triangulation, the remaining three types of triangulation were included in the analysis of this study, and are indicated in the following findings chapters. Regarding investigator triangulation, this study only included one researcher.

Nonetheless, it could be argued that investigator triangulation, or the use of different perspectives to interpret a set of data, can partially be accounted for through peer debriefing. In particular, supervisors of this PhD thesis acted as such during the analytical phase of this inquiry.

Dependability of data is a criterion for trustworthiness in qualitative inquiry that refers to the provision of a clear, thorough and thick description of the data collection process, as well as of the context of inquiry (Lincoln &

Guba, 1985). Techniques to achieve dependable data also include the full description of assumptions made during the research, presentation of a clear audit trail of how data was collected, and triangulation of data. It is argued here that this research study complies with such criteria. Full descriptions, audit trails, and evidence of triangulation of data follow in the next chapters addressing the intervention of this study (Chapter 5), the findings of this study (Chapters 6 to 8), and the discussion of this study (Chapter 9).

Regarding the issue of confirmability of naturalistic inquiry, Lincoln and Guba (1985) describe confirmability as a measure of objectivity for naturalistic research. The principal technique to achieve confirmability is the provision of a thorough and in-depth audit trail of the research. These authors argue that “a single audit, properly managed, can be used to

determine dependability and confirmability simultaneously” (p. 318). As for dependability as discussed above, a full description and audit trail of this research is included in the following chapters.

Finally, the last criterion of trustworthiness of qualitative and naturalistic inquiry indicated by Lincoln and Guba (1985) is the concept of transferability. The positivistic version of transferability of qualitative research is the concept of external validity, which refers to the degree of generalization of the findings of a study (Cohen et al., 2007). Nonetheless, ontological and epistemological approaches of naturalistic inquiry make generalization within naturalism, “in a strict sense”, impossible (Lincoln &

Guba, 1985, p. 316). In naturalistic inquiry the researcher can only set out working hypotheses based on a description of the time and context in which such hypotheses were found. Whether such hypotheses can be found in a different context or time is a matter of empirical research (Lincoln & Guba, 1985). So, instead of specifying the external validity and generalizability of qualitative and naturalistic inquiry, the naturalist researchers can provide the necessary thick description of the contexts and methodology of a study, for someone else interested in such work to decide if findings are transferable into conclusions valid in other settings. A thorough and thick description of the research setting and contexts were described in this study. Whether findings from this study are transferable to other settings can be decided by the reader, and/or investigated in future research.