Validity and Reliability

Determining the validity of a study is necessary to signal that certain considerations have been made to ensure that the research has been carried out in a justifiable way. Validity is akin to addressing mitigating factors in risk assessment; all risks might not be mitigated, but if key issues are covered and addressed, risk is significantly diminished. Validity acts as a kind of mitigation of the risks of research by offering justifications for choices made in the research process. Validity and reliability of RJM was addressed above

through the discussion of the epistemological assumptions, the RJI methods, and the scoring methods, thereby demonstrating construct validity. Kelly (2002) described construct validity as “full and accurate reporting of all aspect of processes of interviewing and analysis to allow examination and

replication” (Construct Validity section, para. 1). In addition to construct validity, RJM has the benefit of a high level of interrater reliability (King & Kitchener, 1994, p. 114). Thus, the remainder of this section will be devoted to discussion validity and reliability in phenomenographic research.

3.6.1 Validity

Åkerlind (2005) discussed two kinds of validity in phenomenographic research, communicative and pragmatic. In communicative validity, the search for defensible interpretation takes precedence over the precise interpretation. This defence can take the form of review and approval by a research community. In the case of this research, communicative validity consists of the research community of my doctoral colleagues and faculty who critiqued this work at University of Liverpool’s Ed.D. thesis residency, the ethics panels both at University of Liverpool and NYU Abu Dhabi to whom I needed to defend and provide rationale for my research design, and my thesis supervisors. The other kind of validity, pragmatic validity, requires the

validation of outcomes that have produced effective ways of operating. Pragmatic validity is not feasible for this study, as the findings have not yet officially been shared with the Office of Global Education. Within the

communicative validity paradigm, the intended audience for the findings can act as a way to measure validity, but it is not necessary.

Ashworth & Lucas (1998) stressed the need for bracketing – that is, the declaration of ones presuppositions about the research being performed – as a necessary part of phenomenographic research on the lifeworld. They recommended that literature searching should take place after analysis to be able to listen without presupposition to what the participant is saying about their lifeworld (p. 421). The presuppositions about theories and the research questions that were brought to this research have been directly addressed in the first chapter, but bear repeating here for validity purposes. Before the data was collected, readings on phenomenography and reflective judgment helped to shape research questions, but most of this work was around developing the methodology for this study and not focused on prejudicing the results.

Regarding the object of study, research on STSA and study abroad was performed after analysis and therefore were not involved in colouring the data collection in any way, which could be considered a unique feature of this study. With respect to the intended discussion on transformative learning, research on TLT was done before the data collection process, and could be seen as an interpretive lens in which I was hoping to discuss the findings.

However, this hypothesis of transformative learning is an interpretive lens on which the research rests. Ashworth & Lucas (2000) admitted that complete suspension of presuppositions through bracketing is very difficult, but it is up to the researcher to be as explicit as possible about choices made, and how bracketing has been achieved (p. 307).

Additionally, Lucas (1998), in discussing the importance of individual profiles in phenomenographic research on lifeworlds, points to these profiles as a kind of internal validity, which refers to the consistency of the

participant’s account. This kind of validity is dependent upon the extent of reflection from the participant, the appropriateness of interview questions, and a lack of trust between the interviewer and the participant (p. 138). For this research, students reflected well on their experience given the interviews were conducted 6-8 months after the trip, and the interview data showed the interview questions and trust with participants did not seem to be an issue.

My position in this research has been presented in the first chapter (page 5), and it is important to reiterate that I have had little to no experience with the Office of Global Education before this research study, yet I was generally aware of the work being done. I made every attempt to suspend my knowledge – however general – of the activities of the Office of Global

Education, and of these regional academic travel trips. Once I engaged with the Office, I had a couple of meetings with the head of the Office, which led me to establish this research project, but I did not receive any other

information from the head on details of these trips.

In sum, validity for the phenomenographic research in this study uses a communicative validity approach, with a reliance on bracketing

presuppositions (including my own) as clearly as possible to ensure choices are justified and research is considered valid.

3.6.2 Reliability

Sandberg (1997) rejected the notion that interjudge reliability in phenomenographic research is defensible (pp. 205-208). The process of relying on multiple readers of phenomenographic research data is flawed in that it shifts the faithful accounting of research participants to multiple and inconsistent perspectives. This is especially troubling in phenomenographic

research that is attempting to utilize phenomenological epistemologies (i.e. lifeworld) since the focus moves from participants’ conception of reality to interjudge reliability. Instead, declaration of the researcher’s interpretative awareness, including all biases toward the data, is important to claim research reliability. Similar to the claims of validity in the previous section, the need to bracket is key in research reliability. Sandberg (Ibid.) suggests using

phenomenological reduction (circumventing certain kinds of pre-definition) in the interpretive space of phenomenographic research. This five-step process entails: a. researcher orientation to the phenomenon as and how it appears, b. always describing the phenomenon in the context of the individual

experience, c. horizontalization, or treating all aspects of the lived experience as equally important, d. a search for structural features, and e. using

intentionality as a correlational rule (p. 210). For this research, the orientation of the researcher is described in the positioning section of the first chapter and kept in mind throughout the entirety of the project; description of the phenomenon in the context of individual experience is a necessary condition of this research given the aims of accessing the learning lifeworld;

horizontalization has been achieved in the analysis by treating each individual account and each response with the same level of importance consistent with treating the phenomenographic data as a whole object; the search for

structural features, as mentioned above, was done through pattern coding once the search for meanings took place in the first cycle of coding the data; and intentionality is built into the analytic process through the initial

construction of an individual profile, followed by the development of themes, and finishing with categories of description.