Reliability

Yin (2009) defines reliability as:

“…demonstrating that the operations of a study – such as data collection procedures – can be repeated with the same results.” (p.34)

Banister et al (1994) suggest that reliability – in a similar way to objectivity – is not appropriate for single case studies. However, Yin (2009) suggests that the goal of reliability is applicable to case studies as it aims to reduce errors and biases in the case study. Therefore he suggests that reliability is enhanced by good documentation of the processes followed so they could be repeated by another researcher. The threats to reliability and the steps taken to control these threats are shown in Table 12 below. Table 14: Threats to reliability and steps taken to control

Threats to reliability Steps taken to control

Reliability and transcripts:

Accurate transcription of responses

ƒ Interviews conducted face-to-face and to be recorded on digital hand-held

recorder, to be transcribed at the earliest opportunity

Threats to reliability Steps taken to control Reliability and transcripts:

Accurate transcription of responses continued

ƒ Transcripts of interviews shared with participants to comment on and check for accuracy

ƒ Full interview recorded and transcribed to avoid selective recording of data Reliability and interviews:

All interviewees to understand the interview schedule in the same way

ƒ The interview schedule was piloted with two participants to ascertain clarity of wording

ƒ Prompts for the interview schedule were written in and were used in a standard way to check understanding. However, a structured interview schedule was deemed inappropriate as this allowed more detailed data

ƒ A semi-structured realistic interview schedule devised to help maintain the sequence of the functions of the interview

Reliability and interviewer:

Good rapport between the interviewer and interviewee

ƒ The function of all of the interviews was explained clearly at the beginning of the interview so interviewees were clear of the reason for the interview

ƒ The researcher is known to the FSWs and Children Centre managers as she has met with them on a number of occasions through her role as an EP

ƒ The interviews were conducted

individually or in pairs, in the preferred context in order to help participants feel comfortable with the process.

Reliability and questions in the interview:

Questions in the interview not to be leading

ƒ The questions were piloted with two participants who were part of a different group that accesses group supervision ƒ The questions were carefully phrased

with prompts to ensure aspects were covered rather than to direct

participant’s responses Reliability and the coding of interview

transcripts:

Coding of participants’ transcripts checked to check it reflected views of participant

ƒ All transcripts coded by same researcher ƒ All coding of transcripts fed back by

email to each participant to check accurate interpretation of context, mechanisms and outcomes

3.5.3 Validity

Robson (1993) defines validity as:

“...concerned with whether the findings are ‘really’ about what they appear to be about.” (p.66)

Yin (2008) outlines three types of validity to consider within validity: construct validity, internal validity and external validity. Construct validity refers to whether the correct operational measures have been selected for the concepts being studied. Yin (2008) notes that construct validity is particularly challenging for case study research and that subjective judgements from individuals are used to collect the data.

Table 15: Threats to construct validity and steps taken to control

Threats to construct validity Steps taken to control Construct validity: Is the research

developing descriptive particulars that exist in practice?

ƒ Use of multiple sources of evidence from a number of FSWs and

Children’s Centre managers, not relying on one single view. They work at different levels within the organisation

ƒ Establish a chain of evidence to show where the descriptive particulars has been derived from

ƒ Have key participants and peers review the draft case study report

Internal validity relates to whether the intervention has actually caused the outcome. Yin (2009) suggests that internal validity has less relevance to exploratory studies, such as in this research, due to the difficulties of being able to cross-check whether the intervention has caused the outcome. However, Silverman (1993) suggests that issues of internal validity are worth considering because, otherwise, research can become little more than descriptive accounts.

Table 16: Threats to internal validity and steps taken to control

Threats to internal validity Steps taken to control A changing population of FSWs who

have engaged in the group supervision

ƒ All FSWs including those who have chosen not to participate have been interviewed in order to understand the reasons why some engage and some may choose not to. This question is also included on the interview schedule

The culture of the Children’s Centres

involved is not fully known ƒ Semi-structured interviews were conducted with a range of participants, Children’s Centres manager and FSWs to allow multiple perspectives to be gained on the impact of the group supervision approach

ƒ The researcher was at all of the preceding group supervision sessions and so was familiar with the culture of those sessions

ƒ The researcher has worked alongside some of the FSWs as an EP and has some understanding of the role and culture of the Children’s Centres ƒ The researcher has some

understanding of Children’s Centres through research and work within Children’s Centres in other roles Changes may occur because of the

impact of the realistic interview schedule ƒ The interviews were held over a short period of time of six weeks to keep the realistic interview schedule consistent

There was one researcher examining the interview transcripts and framework for analysis

ƒ Peers within Educational Psychology Doctorate group support in

examining the data gathered for a short period of time

Other aspects have changed in the FSWs’ environments other than those forming a direct part of the research

ƒ Questions in the semi-structured interview schedule allowed for discussion of a range of factors The findings may not reflect the

participant’s views ƒ Ensuring participants have access to transcripts of their interviews ƒ Ensuring participants agree to the

coding of their interviews and are happy with the interpretation ƒ Making available and feeding back

findings to all participants; FSWs and Children’s Centres managers

Yin (2008) defined external validity as whether the findings are generalisable beyond the research. Again, this is particularly challenging for case studies. However, as Yin (2008) emphasises, this is because the focus should be on analytical generalisation where the researcher is:

“…striving to generalise a particular set of results to some broader theory” (p.36)

Therefore Banister et al (1994) suggest that reflexivity at the personal and professional level as the most appropriate way of gaining external validity in qualitative research. This means the researcher has to reflect upon his/her own bias and create conditions whereby the research can be replicated. Similarly, Pawson & Tilley (1997) encourage researchers to move away from trying to create universal theories but to describe the particulars of a social programme that can be used as the basis of a set of ideas that could be used in an organising framework for future similar social programmes – the descriptive particulars. Therefore it is important that a high level of detail of the context, mechanisms and context are recorded.

Table 17: Threats to external validity and steps taken to control

Threats to external validity Steps taken to control

The research is biased ƒ The researcher constantly examines for personal bias or assumptions within the interviews by checking analysis and interpretation of

transcripts with each interviewee and for biases within the process through discussion with tutor

The research can not be replicated ƒ The researcher keeps a detailed account of the sample, data analysis, procedures and outcomes so this enhances the opportunities for other researchers deciding if the research has applicability elsewhere

Threats to external validity Steps taken to control The research is not built upon a

theoretical base

ƒ The research documents the theoretical bases of Realistic Evaluation explicitly, and draws upon previous research in related fields. This allows further research to understand the methodological framework of the present study and build upon it further