Establishing Validity and Trustworthiness

Merriam (1998) observed that “all research is concerned with producing valid and reliable knowledge in an ethical manner” (p. 198). Eichelberger (1989) pointed out that “what is needed is the confidence that the data are adequate to document, at some level of probability, that the conclusions made in a study are accurate reflections of the relationships among the variables” (p. 232).

3.5.1 Validity

Langenbach et al., (1994) defined validity as “the determination of the credibility of the data being measured or understood in a research study” (p. 377).

Johnson and Christensen (2000) pointed out that “when qualitative researchers speak of research validity, they are usually referring to qualitative research that is plausible, credible, trustworthy, and, therefore defensible” (p. 207). Stake (1995) reported that the question each qualitative researcher must ask is, ‘Did I get it right?’ and pointed out that validity referred to the degree of confidence that can be placed upon the findings of a study.

In addition, there must be consistency in the data collection, analysis, and reporting. Owens (1982) subscribed to this idea by arguing that “naturalistic research is trustworthy when it is accurate or based upon well-corroborated evidence” (p. 16).

Owens went on to suggest:

Validity was enhanced through the strategy of a thick description, and that the purpose of a thick description was to take the reader there…so as to provide a report that yield a rich sense of understanding events and of having insight as to their meaning or more likely, meanings” (p. 17).

In order to help deal with the problem of establishing validity and reliability of qualitative case study research, Yin (1994) outlined three principles of data collection:

1) using multiple sources; 2) creating a case-study data base; and 3) maintaining a chain of evidence (pp. 90–99).

3.5.2 Trustworthiness

Stake (1995) argued that since the establishment of qualitative research within the research realm, the term validity had become somewhat vague and trustworthiness had become more applicable instead and that trustworthiness referred to the overall quality of the research. Langenbach et al., (1994) defined trustworthiness as “the determination that the data are consistent, credible, transferable, dependable, and confirmable” (p. 377). Lincoln and Guba (1990) approached the basic issue of trustworthiness through a series of questions:

How can an inquirer persuade his or her audiences (including self) that the findings of an inquiry are worth paying attention to, worth making account of?

What arguments can be mounted, what criteria involved, what questions asked, what would be persuasive on this issue? (p. 290)

Firestone (as cited in Merriam, 1998) noted that in terms of trustworthiness, the qualitative study “provided the reader with a depiction in enough detail to show that the author’s conclusion make sense”(p. 199).

Merriam (1998) provided three criteria for trustworthiness: internal validity, reliability, and external validity.

3.5.2.1 Internal Validity

Internal validity “deals with the question of how one’s findings match reality”

(Merriam, 1998, p. 201). Mayan (2001) suggested:

To be internally valid, the conclusions of the research must be supported by the data. Internal validity is judged according to the accuracy with which a description of particular event represents the data – complete confidence that your conclusion come from the data. (p. 25)

To ensure internal validity, Merriam (2002) provided basic strategies a researcher can take:

• Triangulation: using multiple investigators, multiple source of data, or multiple methods to confirm the emerging findings in order to lessen the likelihood of error in results, to increase the validity, and ensure that the research results will be dependable

• Member checks: taking data and interpretations back to the people from whom they were derived and asking them if the results are plausible

• Peer review/examination: discussions with colleagues regarding the process of study, the congruency of emerging findings with the raw data, and tentative interpretations

• Researcher’s position or reflexivity: critical self-reflection by the researcher regarding assumptions, worldview, biases, theoretical orientation, and relationship to the study that may affect the investigation

• Adequate engagement in data collection: adequate time spent collecting data such that the data become “saturated”; this may involve seeking discrepant or negative cases of the phenomenon

• Maximum variation: Purposefully seeking variation or diversity in sample selection to allow for a greater range of application of the findings by consumers of the research

• Audit trail: a detailed account of the methods, procedures, and decision points in carrying out the study

• Rich, thick descriptions: providing enough description to contextualize the study such that readers will be able to determine the extent to which their situation matches the research context, and hence, whether findings can be transferred

• Participatory modes: involving participants in all phases of research from conceptualizing the study to writing up the findings. (p. 31)

For this study, several strategies were used to ensure internal validity. For example, in triangulation, three methods were used for the collection of data, which was collected from the five constituent colleges of UNIMA. Member checks were also used as the respondents, for accuracy, verified the transcribed interviews. The three levels of

participants (at Council, University Office and College) provided variation in the findings of the study, such that the resulting data was a ‘rich and thick description.’

3.5.2.2 Reliability and Consistency

Merriam (1998) defined reliability as “the extent to which one’s findings can be replicated” (p. 205). In other words, whether or not the study will produce the same findings if it is repeated. Lanenbach et al., (1994) defined reliability as “an estimate of the degree to which an instrument or other observation will produce similar results time and time again” (p. 374). Miles and Huberman (1994) noted that the underlying issue of reliability was “whether or not the process of the study is consistent, reasonably stable over time and across researchers and methods” (p. 278), and this was not easy in naturalistic studies.

Merriam (1998) observed that in a quantitative research design, reliability “is based on the assumption that there is a single reality, which if studied repeatedly, will give the same results” (p. 205). Since this is a misfit when applied to qualitative research, Lincoln and Guba (1985) suggested thinking about the ‘dependability’ or

‘consistency’ of results obtained from data (p. 288). Lincoln and Guba proposed that rather than demanding that outsiders get the same results, they [outsiders] should concur that, given the data collected, the results make sense – they are consistent and dependable.

Merriam (1998) outlined some of the techniques a researcher can use to ensure the results were dependable:

• investigator’s position: the investigator should explain the assumptions and theory underlying the study, the rationale for selecting particular respondents and the context within which data is obtained

• triangulation: the researcher should use multiple methods of data collection and analysis. The rationale is that this will eliminate the error to a greater degree, thus reliability as well as internal validity can be strengthened

• audit trail: the investigator should document in detail how data were collected, how categories were derived, and how decisions were made throughout the inquiry, for ease of replicating the study. (pp. 206-207)

In this study, the respondents to the study were those who were involved in the management of UNIMA, at either policy, supervisory, or implementation levels. The assumption in this case was that the data collected was dependable. As was mentioned earlier, multiple sources of data in the study strengthened the reliability and internal validity of the study.

3.5.2.3 External Validity

According to Merriam (1998), “external validity [generalizability] is concerned with the extent to which the findings of one study can be applied to other situations”

(p.207). Langenbach et al., (1994) defined generalizability as “an estimate of the degree to which results demonstrated in a research project are generalizable to the population”

(p. 369). Mayan (2001) pointed out that “qualitative inquiry is the degree to which the audience or reader of the report is able to transfer the research findings to contexts outside of the study situation to other settings” (p. 26).

Survey research rests on the assumption that the results of a random sample can be generalized to the entire population that the sample is intended to represent (Gall et al., 2003). However, references to generalizability are of limited use when using the interview method or research. The aim of a single case study research is to relate and perhaps interpret events within a setting (Merriam, 1988, p. 10). Generalizability is limited to the possibility of the researcher supplying “a substantial amount of clear and

detailed information or thick description about the issue/phenomenon studies and the setting in which that issue/phenomenon was found” (Mayan, 2001, p. 26). This assists the reader to decide whether or not the findings are transferable and the degree of transferability, according to Mayan, “is a direct function of the similarity between the two contexts” (p. 26).

Commenting on external validation, Eisenhart and Howe (1992) stated that the researcher must show that “the characteristics of the people, settings, and variables that define the experimental conditions are likely to matter when the treatment is applied to other targeted populations and situations” (p. 645). Merriam (1998) observed that “it is the reader who has to ask, what is there in this study that I can apply to my own situation, and what clearly does not apply?” The onus of concern for generalizability would then be with the reader and not the researcher. Merriam recommended that external validity could be improved by:

• Providing a rich, thick description so that anyone else interested in transferability has a base of information appropriate to the judgement

• Establishing the typicality category of the case

• Conducting a cross-site or cross-case analysis. (p. 211-212)

In this study, interview and documentary data were cross-checked with the respondents throughout the investigation to ensure that the accounts were true and the descriptions were accurate.