Reliability, Validity and Generalisability .1 Reliability

Reliability is ‘the extent to which a test or procedure produces similar results under constant conditions on all occasions’ (Bell, 1999: 103). It relates to the accuracy of methods and techniques used in the research, generally in quantitative research which relies on standardised research instruments, and on cross-checking the data produced by such standardised instruments and by different sets of instruments, which are designed to ‘measure’

the same phenomenon, as a means to check reliability (Mason, 2002: 39). However, the value or feasibility of such standardisation and the concept of research instruments (indicating that such instrument can be neutrally applied) may not always be applicable to qualitative research (Mason, 2002: 39). For instance, the transparency of the whole research process is regarded as important for the quality of the data in qualitative research, which relates to the reliability (of methods) (Seale et al., 2004: 407). Considering the difference between quantitative research

and qualitative research, this research was concerned with reliability in terms of the accuracy of data generation and analysis.

In most qualitative research, reliability (the stability of findings) is improved by triangulation (Gray, 2004: 344). Among different types of triangulation, this study used multiple methods (questionnaires, interviews, observation and document analysis) for data collection. Information gathered through the multiple methods provided a means of testing one source of information against other sources. Both correspondences and discrepancies can be valuable since the same findings from two sources can cross-validate each other, while a discrepancy may be useful in explaining the phenomenon of interest (Robson, 1993: 383). In this research, the investigation of peace education from different people’s viewpoints also increased the reliability of the qualitative data as a form of triangulation in interpretative research (Cohen and Manion, 1989: 270). Overall, triangulation contributed to improving the quality of data and consequently the accuracy of the findings here.

On the other hand, possible causes of unreliability include subject error and subject bias (Robson, 1993: 67). In participant observation carried out here, subject error could occur when pupils’ performance in the project might be affected by internal or external conditions such as their psychological or physical states caused by the school or outside environment.

For this reason, in this research, the observation took place over a sufficient period of time, alongside an attempt to identify sources of error, with a view to ensuring the stability of the findings. As regards the interviews with participants of the project, subject bias can be a problem, as pupils or teachers may try to give answers to please the researcher. In order to reduce the possibility of this bias, the interviewees were encouraged to give honest responses providing useful feedback for the research.

For a case study design, reliability can be achieved if other researchers conducted the same case study, by following the same procedures as described by a researcher, and then reached the same findings and conclusions (Yin, 2003: 37-8). The general suggestion for reducing the

reliability problem is to document the procedures followed by making as many steps as possible, with a view to allowing other researchers to repeat the research. Thus, this study attempted to describe the procedure of the empirical work in as detailed a manner as possible.

Another technique for increasing reliability is to allow the reader to follow the origin of any evidence presented in the research, including initial research questions and conclusions (Yin, 2003: 105). For this aim, the case study report should have made sufficient citation of the relevant portions of the database by citing specific documents, interviews or observations.

The database should present the actual evidence with an indication of the circumstances (e.g.

the time and place) under which the evidence was collected. These issues were taken into account in this case study report when the findings are discussed.

5.4.6.2 Validity

Validity refers to ‘whether an item measures or describes what it is supposed to measure or describe,’ according to Bell (1999: 104). Similarly, for Robson (1993: 66), validity implies

‘whether the findings are “really” about what they appear to be about’. To demonstrate validity, a study should observe or ‘measure’ in the way initially proposed, so that other researchers using the same research instrument can gain the same results (Bell, 1999: 104).

Generally, there are three key types of validity: construct validity, internal validity and external validity (generalisability).

Construct validity is about ‘establishing correct operational measures for the concepts being studied’ (Yin, 2003: 34). In order to increase construct validity, this research used multiple sources of evidence from documents, questionnaires, interviews and observations, with a view to converging lines of inquiry (Yin, 2003: 36), since similar patterns of findings from different data collection methods can increase the validity (Robson, 1993: 69). Another aspect of construct validity is the measurement of abstract concepts and characteristics (Gray, 2004: 91), for example, pupils’ abilities, attitudes and skills relating to peace issues, which were

investigated in this research. Since a measure has to be validated by measuring what should be measured, each of the characteristics of the concept ‘has to be operationally defined before it can be measured’ (Gray, 2004: 92). To establish the validity of the measures in this research, some key concepts which are relevant to the focus of observation and the content of questionnaires and interviews (i.e. the core elements of pupils’ learning in the project, such as affirmation, co-operation and communication) were defined prior to the study.

Internal validity refers to the establishment of a causal relationship and commonly relates to explanatory or causal studies, including causal (or explanatory) case studies (Yin, 2003: 34, 36). Since this research had an explanatory aspect to examine the impact of the project on pupils, questionnaires were used to find possible changes in pupils’ attitudes before and after the project. In the process, there was a concern over a threat to internal validity (Robson, 1993: 70-1). For example, when pupils become familiar with certain ideas relating to questions asked before the project, this may have some influence on their answers to the same questions asked after the project. To stabilise the result of the questionnaires, appropriate occasions needed to be chosen to avoid this subject error. Therefore, in this research, there was a gap of over three months between the first and the second questionnaires, so that the subject error was more likely to be avoided.

5.4.6.3 Generalisability

In carrying out case study research, the issue of generalisability is a major barrier, regarding the extent to whether the findings can be generalised beyond the immediate case study, in particular, in single cases (Yin, 2003: 37). Generalisability refers to ‘the degree to which findings can be generalized from the specific sample in the study to some target population’

(Robson, 1993: 46), or ‘in other contexts, situations or times’ (ibid: 66). While threats to generalisability can be related to selection, setting or period (when the findings are only relevant to the group studied, to the particular context in which the study was carried out, or to

specific historical experiences) (Robson, 1993: 73), there are two main strategies to reduce these potential threats: first, ‘direct demonstration’, in which the results are applied to a further study (e.g. with other types of participants or in a different setting); second, ‘making a case’, which justifies the generalisability of the results, ‘with arguments that the group studied, or setting, or period is representative in that it shares certain essential characteristics with other groups, settings or periods’ (ibid: 72). Since the peace education project studied in this research focuses on improving interpersonal skills (co-operation, communication, affirmation and problem-solving skills), which are typical skills in peace education, it might be possible to make some kind of generalisation about similar types of peace education.

Moreover, generalisability in case study research can be distinguished from survey research which relies on statistical generalisation, whereas case studies rely on analytical generalisation in which the researcher intends to ‘generalize a particular set of results to some broader theory’ (Yin, 2003: 37). Thus, ‘the findings of qualitative research are to generalize to theory rather than to populations’ (Bryman, 2004: 285). It means that generalisation is assessed by the theoretical inferences made from qualitative data. Following this idea of generalisation, in this study of a peace education organisation and its project in a school, the data was analysed to connect with and generalise to wider theories of peace, peace education or schooling.