V ALIDITY

Validity reflects the proximity of the researcher’s conclusions to reality, so validity is relative and must be assessed in relationship to the purpose and circumstances of the research (Maxwell, 2005). In this study, the question of validity concerned whether the research could represent the cases in a “true” way. A key concept for validity is the “validity threat”, and Maxwell (2005) claimed that validity is made implausible by evidence, not methods. Methods are only a way of obtaining evidence that can help to rule out these threats (Maxwell, 2005).

According to Maxwell (2005), there are two broad types of threats to validity, which often arise in relation to qualitative studies: researcher bias and reactivity. Researcher bias concerns the selection of data that fit the researcher’s existing theory and the selection of data that

“stand out” to the researcher (Miles & Huberman, 1994), and involves the subjectivity of the researcher (Maxwell, 2005). “Reactivity” refers to the influence of the researcher on the setting or on the individuals studied. It is impossible to eliminate the actual influence of the researcher, and the goal of a qualitative study is not to eliminate this influence, but to understand it and use it productively (Maxwell, 2005). Although specific procedures do not guarantee validity, they are essential to the process of ruling out validity threats and increasing the credibility of the research conclusions. Such procedures do not operate by

verifying conclusions, but by testing the validity of the conclusions and the existence of potential threats to those conclusions (Maxwell, 2005).

Lack of “rich” data may be a validity threat (Maxwell, 2005). In this study, the data were collected in such a way that they were detailed and varied, to provide a full picture of the implementation of the national guidelines at the case schools. A pilot observational study and a pilot test comprising interviews with a principal, a focus group of teachers, and a focus group of students were conducted to assess the usefulness of the observation form and the interview guides. The observation form was made more extensive after the pilot test. The participants in the pilot test reported that the questions were relevant, and only proposed changing the order of some questions. The focus group interviews with the teachers and students elicited different viewpoints. Kvale and Brinkman (2009) claim that focus group interviews are well suited for exploratory studies because the lively collective interaction may bring forth more spontaneous expressive and emotional views than do individual interviews.

Two researchers participated in the data collection across the case schools and this may have ensured that all the relevant information was noted. Furthermore, the use of an observation form and interview guides, audiotaping the interviews, and transcribing the interviews verbatim contributed to the collection of detailed, specific, and descriptive data.

Maxwell (2005) claimed that triangulation reduces the risk of chance associations and the systematic bias attributable to a specific method, and allows a better assessment of the generality of explanations. In this study, triangulation was achieved by comparing the data from interviews with the data obtained from the observations, baseline questionnaires, and documents. The data from the principals were compared with those from the teachers, students, and project leaders. Respondent validation was also assessed by examining the

patterns of data collection. Observational data were collected before the interviews. This allowed the researchers to obtain additional information that was missed in the observations, and it was used to check the accuracy of the observations. The observations gave the researchers an understanding of the participants’ perspectives that were reported in the interviews.

Explicit comparisons, used to assess the validity threats, are used in multiple-case studies (Maxwell, 2005). In this study, the data were compared and contrasted between the

participants and case schools, and three researchers discussed the analyses to ensure that the intended meaning of the data was captured.

According to Yin (2003), there are two tests for the validity of a case study with an exploratory design: construct validity and external validity. Construct validity concerns establishing the correct operational measures for the concepts being studied. In this study, construct validity was assured by using multiple sources of evidence, so that it was possible to triangulate the data. Triangulation was achieved by comparing the data from interviews with the data from observations, baseline questionnaires, and documents. The data from the principals were compared with those from the teachers, students, and project leaders. The assessment of external validity involves establishing the domain to which the study’s findings can be generalized and is evaluated with replication logic (Yin, 2003). This includes a full description of the cases, which permits adequate comparisons with other samples, a full description of the sampling strategy, and an explicit definition of the theoretical framework.

Kvale and Brinkmann (2009) describe three forms of generalization. Naturalistic

generalization is based on personal experience and personal or individual knowledge, and leads to expectations rather than predictions about assumed associations. Statistical generalization is based on representative subjects selected at random from a population, to allow generalization to a larger population. Analytical generalization is a reasoned judgment about whether the findings from one study or case are applicable to another, based on similarities and differences between the studies or cases. Rather than a naturalistic

generalization, the generalization claims of the study are based on an ascertainable logic by making the arguments explicit. The overall aim of this study was not to achieve statistical generalizability but to look for similarities and differences within and between cases, and to compare and contrast them. Therefore, analytical generalization is applicable.