Validity and Reliability

The meaning of validity and reliability in qualitative research as well as the extent to which it can and should be pursued is debated. Qualitative researchers frequently use alternate

terminology such as “credibility”, “trustworthiness”, or “dependability” to refer to the quality of research (Lincoln & Guba, 1985). The validity of qualitative research, defined here as the extent

to which it measures what it set out to measure, can be ensured through a number of quality control mechanisms and checks pertaining to the various research steps and stages where research outcomes might be affected by the decisions or processes employed. One of these is to adopt rigorous analytic methods. Cope (2004) argues for the use of the phenomenographic structure of awareness as a framework for data analysis to this end, and I have elected to do so in this research. The key validity question for this study is whether I have accurately and

thoroughly captured faculty member experiences of the academic library. Two styles of validity check which can be applied to this question and which are commonly used in

phenomenographic inquiry are termed communicative and pragmatic validity (Akerlind, 2005). Methods specific to these styles and their application in this study are discussed in this section. A primary threat to the validity, or perceived validity, of qualitative research in general is misinterpretation of data by the researcher, and this can be addressed with communicative checks. The challenge is that many legitimate, alternate interpretations of data and

phenomenon may exist in qualitative research – and therefore the correctness of an interpretation lies in its ability to be persuasively defended to the relevant audiences.

Communicative validity checks present such defences and arguments to relevant audiences and gather feedback via peer review, conference presentations, and other formal review

mechanisms, as well as feedback from members of the population that the research sample represents and the audience for the research findings. The technique of member-checking, well-known in qualitative analysis, is another form of communicative validity check where the research participants themselves are asked for feedback on the researcher’s interpretations. However, this technique has criticisms both from the realm of phenomenography and in qualitative research more broadly. Oliver (2003) gives voice to several such criticisms, including the argument that participants who are untrained in social science methods would be ill-

equipped to ‘check’ the data or analysis of a researcher in any meaningful way. These checks, according to this argument, may not aid in verifying the accuracy of the conclusions as much as the satisfaction of the participant with their own self-presentation or desired conclusions. Akerlind (2005) argues that this type of validity check is particularly inappropriate for

an individual level, and the meaning and significance of an individual interview transcript cannot be understood without the context of all the others in a data set. I have elected to forego formal member-checking of this sort and to pursue other communicative validity checks. In the context of the production of a doctoral thesis, I believe that a large degree of

communicative validity can be sought and demonstrated through the persuasive defence of my methods and interpretations to research supervisors and advisors charged with reviewing the work, as well as in conference-style feedback from my peers in both research and practice, both of which ensure that my interpretations are defensible to any other reasonable observer. Pragmatic validity (Akerlind, 2005) focuses on ensuring that the outcomes of research are useful and practical. The development of practical insights as a core aim of phenomenographic research is emphasized by Entwistle (1997) and Bruce (1999), and this emphasis is likewise of great importance in the arena of the professional doctorate. In addition to asking, “has this research measured what it set out to measure?” we can ask, “has this research produced the kinds of practical insights it was meant to produce?” In the case of my inquiry, the practical intention has been to find meaningful directions for the development and evaluation of academic library services to faculty members, and to a large degree then its validity relies on the meaningfulness and utility of the practical recommendations outlined in the results and discussion. The critical judgment of LIS practitioners and faculty member users of academic library services provides evidence in this regard. Lincoln & Guba (1985) approach this from the idea of “transferability”, referring to how the readers of research can identify and extract useful elements of findings and apply them to their own contexts. According to Lincoln & Guba, a procedure which enables this is the use of what they term “thick description”, as opposed to superficial description, which should capture a phenomenon in sufficient detail that those details can be identified and compared to other contexts by readers. While this research certainly employs a rich and detailed structure for descriptions of experience, it also seeks to describe much of the context for the research in the higher education, disciplinary and international branch campus setting. To aid the readers of this research in drawing their attention to relevant details which are likely to affect faculty member experiences with the academic library, the theoretical framework of information worlds (Jaeger & Burnett, 2010) is

put forward as a guide for thinking about and transferring the findings of this research to other settings.

Reliability in qualitative research generally depends on ensuring that the same data would be interpreted in the same or similar way by another reasonable, qualified observer. Would another qualified phenomenographic researcher, given the same data set, reach similar conclusions in terms of categories of description and understandings of meaning? Some researchers rely on dialogue with colleagues and/or multiple data coders to arrive at some consensus and confidence in the reliability of the results (Akerlind, 2005); however, others argue that this is both ineffective and unnecessary because it is rooted in a search for some kind of objective reality; a proposition which has already been rejected from the outset of phenomenographic inquiry (Sandberg, 1997). For the purposes of this research, I would argue that a diligent approach to critical reflexivity, bracketing off my own presuppositions, and offering clarity and transparency in these steps to readers as they are manifested in specific data gathering and analysis activities, will adequately ensure to my own and others’ satisfaction that the results are as reliable, in the sense of dependable, as can be expected in this type of research, while acknowledging the position that a researcher can never be removed entirely from influencing the outcomes of qualitative inquiry. Sandberg terms this approach

“interpretive awareness”, arguing that we must account for the impact of subjectivity in research rather than trying to get around it (1997). In the case of this research, for example, I have shared my position as an academic library practitioner with knowledge of library roles and identity from the practitioner/administrator perspective. Coming from this position, I have identified several of my own, taken-for-granted assumptions about the academic library; for instance, that it is a valuable academic support, and that it serves meaningful roles as a support for faculty teaching and research. My awareness of these “knowns” allows me to adopt an opposite assumption while reading and analysing data, for instance, actively looking for evidence of areas where the academic library is not used or not valuable. Readers must take into account both my position as a practitioner-researcher and my described good faith efforts to move towards an honest representation of participant experiences, rather than my own, when considering the meaning of the findings presented here.

Some special considerations around validity and reliability exist for phenomenographic approaches to research. Marton and Booth (1997) posit that the quality of an outcome space can be judged primarily on how well the categories of description capture distinctive variations in experience, to what extent the categories are logically related, and to what extent the categories efficiently capture critical variations with the minimum number of categories necessary. The data analysis procedures described in detail above, as well as the explanations of categories of experience presented in the results and discussion section, therefore

demonstrate these qualities, in part by describing how the structure of awareness framework was employed in analysis.