Validity and reliability

The design of a research study has to address the criteria of validity and reliability in order to ensure the quality of the research findings. It is not just about justifying that the research is legitimate, accurate and lacks bias. It is also necessary to provide standards and rigor (Miles and Huberman, 1994; Patton, 2002). This corresponds to the question, ‘How can the inquirer persuade his or her audience that the research findings of an inquiry are worth paying attention to?’ (Lincoln and Guba, 1985: 290).

Although reliability and validity are tools of an essentially positivist epistemology, Patton (2002) suggests that any qualitative researcher should be concerned about them while designing a study, analyzing results and judging the quality of the study.

However, underlying assumptions of social constructionism make conventional notions of validity and reliability problematic. Healy and Perry (2000) assert that the quality of a study in each paradigm (positivist or social constructionist) should be judged by its own paradigm’s terms. For example, reliability and validity are essential criteria for quality in quantitative paradigms. In qualitative paradigms, the terms credibility, neutrality confirmability, consistency, dependability, applicability or transferability are to be the essential criteria for quality (Lincoln and Guba, 1985). Strategies need to be built into the qualitative research process to ensure a set of findings is rigorous. A continuous process of checking, confirming, making sure and being certain, was used during the process of this research to incrementally contribute to ensuring reliability and validity and thus rigor of the study.

Validity determines whether the research measures that which it was intended to measure. In other words, do the findings make sense and are they credible to the reader? Connelly and Clandinin (1990) suggest that interpretative studies generate understandings and as such should have an apparency (clearly understood), authenticity and plausibility. Kvale (1989b) emphasizes validity as a process of checking, questioning and theorising. Miles and Huberman (1994) suggest guideline questions such as: how context rich and thick are the descriptions? Does the account ring true and make sense? Are the data well linked to prior or emerging theory? Did any triangulation of data take place? Is the account a comprehensive one? These questions enhance internal validity. However, the researcher needs to attend to the vexed question of external validity. In other words, are the conclusions transferable and can they be generalized?

The literature identifies four levels of generalisation: from sample to population; analytic or theory-connected; case-to-case transfer and theoretical validity - where a more abstract explanation of described actions and interpreted meanings are present.

Questions such as: is the sample sufficiently described to permit comparisons with other samples? Are the findings congruent with, connected to or confirmatory of prior theory? Have the findings been replicated in other studies? Finally, questions related to pragmatic validity (Kvale, 1989a) must be addressed. The ultimate test of the credibility of a study’s findings and hence its validity is the response of decision makers and information users. For example, a piece of research that informs policy makers in society can lead to more intelligent action with real people’s lives being affected (Patton, 1990). Good, qualitative research enhances levels of understanding and sophistication and enables stakeholders to take action (Miles and Huberman, 1994).

Fruitful probes should focus on how accessible the findings are to potential users; what the level of usable knowledge is; and how the findings can stimulate action to solve a problem.

Reliability is the extent to which the findings can be replicated or repeated to give the same results. If the results of a study can be reproduced under a similar methodology the research is considered reliable. Miles and Huberman (1994) distinguished between external reliability or objectivity and dependability or auditability. External reliability questions whether the research is free of unacknowledged researcher biases and whether the inevitable biases of the researcher have been made explicit and managed throughout the study. As Guba and Lincoln (1981) suggest, the study needs to depend on the subjects and conditions of the inquiry rather than the researcher, to achieve external reliability.

Miles and Huberman (1994; 278) offer a set of questions the researcher should regularly ask of the study such as: Are the study’s general methods and procedures described explicitly and in detail? Can we follow the actual sequence of how data were collected, processed, condensed and displayed for drawing specific conclusions? Are the conclusions explicitly linked with exhibits of condensed data? Lincoln and Guba (1985) define dependability as a notion closely corresponding to reliability and emphasise an inquiry audit as one measure, which may enhance dependability.

Dependability is concerned with consistency and stability over time. These can be addressed through inquiry about clarity of research questions and alignment of research design, when the steps of the research are verified through examination of

such items as raw data, data reduction products and process notes (Miles and Huberman, 1994).

In this study, the guideline questions of Miles and Huberman (1994) were attended to throughout the course of study design, data collection and analysis. The researcher has already acknowledged that the ‘snowballing techniques’ used in forming a sample group, compromised the validity of the study and raised the possibility of random errors affecting the findings. The sample of nine individuals for the pilot group included representatives from all three key stakeholders in the appointment process, which resulted in rich triangulated data. The data provided the researcher with thick descriptions from each of the stakeholders and enabled her to check, compare and contrast the different versions of the story from the different stakeholders. This resulted in a robust and holistic description of the appointment process. The findings of the main study are specific to the context of 15 first-time FTSE 100 NEDs and due to the small sample size generalizability beyond their context is limited. However, this chapter includes detailed information about the sample groups for both the pilot and main study to allow comparison with other samples and to provide context for the study. In this chapter the researcher has provided an audit trail for study design, data collection and analysis and has been rigorous and transparent in her use of analytical techniques, such as template analysis, thereby attending to reliability.

The researcher has adopted a reflexive approach throughout the project to enable her to consider questions of reliability and in particular to reduce bias. Data collection from interviews were recorded and professionally transcribed to reduce personal bias in transcription and to stay close to the language of the interviewees, in line with the researcher’s ontological assumptions. In analysis, an iterative rather than a linear process was adopted. The researcher moved backwards and forwards between interview data, analysis and interpretation. Through the mechanisms above, data was systematically checked, focus was maintained and the fit of data and the conceptual work of analysis and interpretation was monitored and repeatedly confirmed. Collecting and analyzing data concurrently allowed the researcher to establish what was known and what she needed to know. The iterative interaction between data and analysis enhanced the validity and reliability. In using a grounded approach to her analysis, the researcher identified themes in individual interview data and across interview data. This iterative process enabled the researcher to move from a micro perspective about the

data to a more macro conceptual and theoretical understanding, inching forwards without making cognitive leaps.