Reliability, Validity and Generalisability 1 Reliability

The reliability of a replication of a qualitative research is by nature low and it is agreed that proposals for the replication of case studies are unrealistic for reasons connected with this type of research (Dey, 1993). In qualitative research it is usually impossible to expect that other researchers in similar or even identical situations can replicate exactly the findings of any particular research project (Schofield, 1989; Merrick, 1999). Thus the basis for the determination of reliability of a qualitative analysis is provided by the overt exposure of the conceptual perspective and the criteria of the researcher who conducted the research. In this research, the researcher revealed to her colleagues and readers how the research was conducted and how she made her decisions, so that readers can judge the research quality and test its logic and the reasonability of the researcher's thinking and actions (Arksey & Knight, 1999). Mason (1996) notes that in order to test the reliability of the research with its unique conceptual context and perspective, an explanation of some sort must be provided concerning the way in which the level of reliability and accuracy was attained.

The only place in which it was obvious that there was a high level of reliability was with regard to the recording and transcription of the interviews. The recording captured the exact words. In order to analyse the wording in the most accurate manner the speech was transformed by the researcher into written text (Stake, 1995; Seidman, 1998). Reinforcement of reliability was also obtained by the fullest possible records of the data (Yin, 2003) and by using the formal protocols which were produced in the research arena. The recorded protocols helped to ensure that phenomena which were revealed as exhibiting clear patterns were actually like this.

121

3.8.2 Validity

Validation means going back and forth between deduction and induction, between experience and reflection concerning the experience, between the data and the conceptual perspective of the researcher and the conclusions. What the researcher discovered must be confirmed by going back to the studied empirical world and examining the extent to which the developing analysis fits the phenomenon that was observed (Patton, 1990). Kirk and Miller (1986) noted that there is no one technique in order to validate a study, apart from a continuous personal interaction. We can never be completely convinced that we understand all the cultural meanings of each studied phenomenon. In this research the researcher tried to be sensitive and wise, and was equipped with a theoretical orientation and close relations and good faith (with the informants) over time, a systematic work model and pattern of behaviour that served as an effective tool for a thorough examination by the researcher.

The researcher was careful to maintain the chain of evidence for each step that she took in the research. She maintained all the transcriptions of the interviews and all other relevant material (Huberman & Miles, 1994). She kept al the documents, memos, analytic 'trees' and flow-charts which were made concerning the analytical process. The chain of evidence is composed of sections of data, in their various versions, from the raw stage of data-collection to the different stages of analysis and until the final findings and conclusions (Yin, 1981).

Maintaining the documentation for each stage of the analysis also enabled the researcher to cast doubt on herself and on her judgment and enabled her to test to what extent she was faithful to the data which were received from the informants (Strauss & Corbin, 1991). Mason (1996) emphasises that the validity of method and of the interpretation must be proved by a meticulous process of examination and restructuring of the path, by which the researcher thinks that the interpretation was reached, and there is no easy answer or short-cut in this process.

In qualitative research instead of the requirement for validity of prediction there is a demand for 'construct validity' (Miles & Huberman, 1994; Yin, 2003, 2008).

122

This relates to the extent of compatibility between the criteria chosen to represent the phenomenon and the concepts to which the criteria relate. In this research, the researcher used various sources and relied on a chain of evidence at the stage of data-collection and writing-up the thesis. She demonstrated that the meaning that she extracted from the studied situation, from reports and from reading had construct validity (Stake, 2000; Yin, 2003, 2008).

When the researcher presented the findings and her interpretation of the case study she took care, with regard to the construct validity, to show that the interpretations that she attributed to the patterns that she discovered, i.e. the second-level structures, were indeed congruent with the first-level knowledge which was collected during the research. Validity was attained when it was possible to demonstrate that the explanation was congruent with the meaning through which the participants constructed their reality and conducted their daily activities (Philipson, 1972). Validity was thus complete when the researcher's interpretations were validated and also when her understandings and interpretations of the studied reality – presented as 'second-level structures' – stood in congruence with the field data and with the research literature.

Qualitative researchers distinguish between 'internal validity' and 'external validity'. Schofield (1989) notes that, if other researchers who read their research data, sense that the evidence does not support their description of the situation then qualitative researchers should place serious doubt on the internal validity of their work. For this reason, the researcher maintained the entire chain of evidence of data and documents and took the advice of colleagues by presenting the data to them in a manner that enabled them to confirm (or place doubt) on the validity of the research. This process was conducted in an informal manner, but in a structured form (Merriam, 1985; 1998; Merrick, 1999).

123

Riesman (1993) indicates that in order for the process to be effective it needs to comply with the following conditions:

1. To make the information available for other researchers.

2. To make the research process with all its components manifest.

3. To provide a description of the way in which the interpretations were formed.

Finally Riessman (1993) suggests that the ability to convince others of the validity of a research is strengthened when the theoretical claims are supported by evidence from the informants' explanations.

'Internal validity' relates to the extent to which the explanation for an observed action is proved as correct in the studied context and it also relates to the question: to what extent are the chosen criteria indeed suitable to represent the phenomenon? (Yin, 2003). Additional internal validation is attained when the interpretation of the researcher receives the confirmation of the informants, in other words the findings are logical and acceptable for the informants (Miles & Huberman, 1994; Maykut & Morehouse, 1994). At the end of the research, the researcher presented the research to the informants before its publication. This contributed to the 'trustworthiness' of the research (Lincoln & Guba, 1985). Lincoln & Guba (1985) call internal validity – 'trustworthiness'.

'External validity' relates to the specific data-collection through which it is possible to arrive at generalisations that indicate a theoretical relationship (Yin, 2003). External validity is an additional stage in the validation process. It is used for cross-checking research findings with the findings from the professional literature, examining the existence of patterns of activity of others, and examining the generalisations made by the researcher. In the present study, the researcher explained her understandings to the reader through excerpts and quotations from the 'thick description'. In this way the researcher laid out the reality in which she conducted the study before her readers and her understandings and interpretations of this reality. Exposure of the findings shows the reader to what extent the researcher was 'close' to the informants and to the data that was gathered and the readers are also given space to

124

provide additional validation in accord with their own personal understanding or familiarity with the studied subject.

Validity was obtained for the interview when in a number of interviews the interviewee consistently repeated the same things in the syntax, made the same pauses in speech, and the same word clusters while the interviewer remained inactive. Thus, the researcher knew that interviewee was trustworthy (Seidman, 1998).

The extent of patterning that was revealed increased the validity of the findings and it was only revealed after the cross-checking of data from different sources. These patterns corresponded with the researcher's generalisations and interpretations which were presented as 'second-level structures'. When the researcher was required to reach a final synthesis, she identified certain 'key events' (Fetterman, 1989).

3.8.3 Generalisation

In qualitative research it is difficult to generalise from one research to another (Yin, 2003). This means that it is very important to pay attention to the context in which the research takes place, in order to examine whether it is possible to generalise the research findings to other similar times, places and populations (Shavelson & Towne, 2002).

There are researchers who believe that it is impossible to generalise qualitative findings and they see this as a limitation of this method (Merriam, 1985; Firestone, 1993). One of the most frequent criticisms concerning qualitative research voiced by quantitative researchers is that qualitative data-collection produces data that cannot be compared because not all the informants are asked the same questions in the interview (Strauss & Corbin, 1991). In response to this, Stake (1995) claimed that the true business of a case study is its uniqueness and not generalisation. The researcher in the present research took a particular case and studied it in order to know it well. Not how it differed from other cases, but rather what were its characteristics and its meanings. By

125

learning the specific qualities of the individual case, it is possible to understand the universal case (Simons. 1996).

Guba and Lincoln (1989) fundamentally reject the benefit of the concept of generalisation and external validity and claim that it should be abandoned as a research goal and replaced by the term 'transferability'. Stake (1995) claims that the qualitative researcher can organise the research so that most of the opportunities for naturalist generalisation can be attained. Accepting the path to generalisation as suggested by Stake (1978; 1995) leads to the conclusion that the consumer of research, the reader and not the researcher is the one who determines the extent to which the research can be generalised and so the onus of proof in qualitative generalisation is imposed less on the researcher and more on the reader (Marshall & Roseman, 1989; Peshkin, 1993; Firestone, 1993).