Reliability and validity

Both reliability and validity are important features of a research study as they represent how consistent and replicable the study is and to what degree the research tools employed have measured what they claim to have measured. The reliability of a research project is concerned with how consistent and replicable the studies and the findings are. Validity, on the other hand, refers to the extent to which the research tools employed in a study provide the data that they are intended to. Without being valid a study, by definition, has no value, and if the measures are not reliable then we cannot answer the research question(s). In the present study, with a realist perspective on the CBT intervention, the focus is on providing a valid account of a phenomenon and is less concerned with providing law-like certainty. In addition, Cohen et al. (2003) argue that in qualitative research the notion of validity should be seen as a matter of degree rather than as an absolute. The present study employs the term ‘reliability’ to refer to the exploration of as broad and as deep an understanding of the processes of CBT as possible, adapted for an AS and delivered in school. A stated premise of the current study is that the contextual features of a school setting may influence the course, nature and outcomes of a CBT programme.

With regard to validity we can make a distinction between the internal and external validity of a given study (Cohen et al., 2003). Internal validity refers to the explanation of a phenomenon being consistent with the data that is used to describe it. External validity is more to do with the degree to which the study outcomes and findings can be generalised to other situations. In the current study a threat to internal validity could be the changes in B’s behaviour as reported on the ASD profile in the pre and post intervention phases, as it relies on observer reports rather than direct observation. This threat was reduced by comparing

126

the observations of more than one observer and recording the degree of concordance between them. It was also possible to compare a section of the profile, ‘sensory difficulties’ which, it was predicted, would not change in the pre and post measures. Both observers reported their observations independently of each other but used the same rating scale. The use of observers who have commissioned the intervention also seeks to increases the social validity of the study, in that it recognises the views of those who have initially raised the concerns and reflects what has been termed customer satisfaction (Kazdin, 1977). Another aspect of validity in the present study is the reflexivity of the researcher or the awareness of the ways in which the researcher’s background attitude and approach influence the research process. In the current study the researcher entered the social system of the school both as researcher and as the school’s existing EP, thus the attitude and approaches of the researcher and the perceptions of those in the school of the CBT intervention all form part of process and outcome of the intervention.

External validity refers to the degree to which a study may be generalised to a situation beyond the context in which the study is currently conducted. The present study considers the possibility that the school context may influence the outcomes of a CBT programme delivered in school. By employing an RE design it is proposed that any observed outcomes of the CBT intervention can be expressed as a configuration of the school context and any triggered mechanisms in that setting. The present study has a highly specific setting and context for the CBT intervention and this represents a threat to the potential generalisability of the findings. The notion of external validity is an important one, since a single participant case study design can be critiqued on the basis that it may not be representative and therefore have limited generalisability and, by implication utility. However, experienced practitioners are able to make explicit comparisons between the school settings in the area in

127

which they work, the nature of therapeutic interventions and different types of pupil. For example, EPs know that on their ‘patch’ (of schools) even if an intervention works in school A it may well not work in school B, but comparison would suggest it may work in school C. Therefore, rather than generalisation of findings, Lincoln and Guba’s (2000) concept of transferability is used in the current study, in which research findings are always viewed as ‘working hypotheses’ rather than ‘facts’, arguing that we can aggregate data from similar settings. Thus a single participant case design can be the basis for designing interventions in different settings, and thus increases the potential utility of the findings of the current study for EPs and practitioners in school.

In the present study a potential threat to validity may have been introduced during the collection of data, with the use of interview methodology. This represents a potential threat to validity in that interviews are researcher led and also represent a social encounter between the researcher and the researched. Various social and cultural influences such as status, social class, gender differences and power relations that may therefore potentially bias the interviewee’s responses to the questions and thus the reported outcomes of the interview. However in the present study a realistic interview technique was employed, that rejects the traditional approaches to interviewing which are underpinned by an assumption that:

… formal structured interview, the informal, open ended interview, the semi-structured interview the multi method approach to data collection are all data-driven strategies … they are thus all constructed under the working assumption that the subject and the subject matter of the interview are one and the same thing.

(Pawson and Tilley, 1997, p. 155 emphasis in the original)

In a realist interview the subject matter of the interview is the theory of the researcher or what mechanisms might operate to bring about an outcome the interviewee can validate or

128

invalidate the theory presented to them. So the views of the stakeholder are captured as they express their perspective on the theory presented to them, in this case as to whether they feel the delivery of a CBT programme for the pupil would be successful in their school. A potential threat to validity, as respondents are reporting their views to the researcher, could be that they give answers they feel the researcher would like to hear. However, a reflexive view of the researcher’s existing relationship with the school suggests that this threat would be much reduced. ‘Reflexivity’ in research means acknowledging that the researcher brings with them an identity, both professional and social, that can influence the course and nature of a study (Robson, 2003). The researcher’s existing relationship with the school in the present study was more of an advantage than a threat to validity. This is because being familiar with, and to, school staff may have militated against what Lincoln and Guba (1985) refer to as ‘reactivity’. Reactivity is the notion that the researcher’s presence in the researched community influences the behaviour of those involved, so that their behaviour, as recorded by the researcher, could be argued to be untypical. However, given ongoing work with the school this is less likely to influence the reactivity of school staff because they are used to the researcher’s presence in school and are used to expressing their opinion. Thus it can be argued that being the researcher and the EP is likely to reduce the potential for bias, since respondents may have felt more comfortable reporting their views, critical or otherwise, rather than saying what they thought was wanted.

Both the ASD profile and the ASD audit were piloted prior to the present case study, with the aim of refining any problematic aspects of the format in terms of language, number of questions and type of rating scale. There was a preference in the piloting phase for the ASD profile in an A4 format; it had initially been in an A3 format. On the ASD audit document respondents preferred a descriptive scale and more space to express their own interpretation

129

and evidence to expand the points raised. A potential threat to validity in a small case study is that the small number of respondents could represent a skewed sample. However, the present study was more concerned with recording what respondents could see in B’s behaviour, and considering the degree to which this matched other respondents’ views, than it was with statistically significant samples. In addition it aimed to provide an account of the process and contextual influences of the school setting and mechanisms that underpin the CBT intervention, rather than just considering the observed outcomes. The data obtained from the different research tools will be triangulated to provide different perspectives on the CBT process and outcomes. Triangulation of data also actualised the retroduction in the realist analysis of the present study and provides a degree of reliability and validity of the measures employed.