Data Collection Tools

Data collection tools were designed to gather relevant data through multiple sources of evidence according to an effective case study design (Yin, 2009). Pawson and Tilley (1997) proposed that any method of gathering data is acceptable within a RE, as long as the tools are sufficient to gather relevant data which enable hypotheses raised in the RS about how a programme works to be tested. Although data gathering tools were multiple and designed to address the research questions, they are not without limitations, as discussed henceforth.

5.6.2.1 Observations

Multiple narrative observations were undertaken in order to focus deductively on the hypotheses raised in the RS, whilst also being open to inductive interpretation of new codes and themes (Braun and Clarke, 2006). Both groups (Year 10 and Year 11s) were observed for exactly the same amount of time (two days each) in order to draw equally from the pupil participants involved and avoid observer bias. It is recognised that using narrative tools means that a different observer would have been unlikely to have observed exactly the same behaviours as were recorded, so validity of the data may be

compromised. However, this could also happen if structured techniques were used and it was felt more important to be open to new codes and themes emerging.

To limit the effects of observer selective memory, observations were written at the time and then coded shortly (within a week) afterwards. The quality of these observations would have been improved if more than one researcher had conducted observations, however this was beyond the scope of this study due to already demanding a high level of Forest School staff time for interviews. Further research into Forest School might wish to draw on both narrative and structured observation tools and observe towards the start of the young people’s Forest School experience as well as later on in order to draw some comparative observation data.

5.6.2.2 Documentary Evidence

This method of data gathering was useful to programme specification development because it set out some elements of how the programme works (e.g. through calculated risk assessments) and what skills young people could be expected to gain through attending (e.g. NOCN, 2013 assessment criteria). However, it is acknowledged that evidence from the documents should be checked via observational data as otherwise there is no guarantee that what is proposed by the documents is actually happening. Therefore, documentary analysis was used as part of triangulated data gathering but not as a sole source of evidence. Evidence from available documents (appendix 8.9) contributed to CMOC development in the final programme specification (appendix 8.19) but was only included when data to support it had also been gained from other sources.

5.6.2.3 Questionnaires

Pawson (2006) suggested that some quantitative data can be used to support the understanding of programme outcomes and therefore a questionnaire was designed to attempt to gather this evidence. Programmes within education are often evaluated after they have started running (Cohen, Manion and Morrison, 2011) and therefore no pre- intervention data was available for this case study group. This weakens the strength of the questionnaire measure, which asked adults who knew the young people well to consider how certain aspects of their development may have been impacted upon by the Forest School programme. In order to avoid leading participants to state positive

responses (Cohen, Manion and Morrison, 2011), options to record ‘no change’ ‘deterioration’ and ‘don’t know’ were offered.

Questionnaire data was used to gain some quantitative understanding of the level of change caused by Forest School, according to adults who knew the pupil participants well. Due to the retrospective nature of questionnaire administration, this data was not drawn upon in isolation to develop CMOCs as this may have reduced the validity of the findings. Instead, questionnaire data was used alongside other data collection tools to support CMOC development. Questionnaires enabled hypotheses about the outcomes of Forest School raised in the RS to be explicitly tested, such as the impact on the young person’s understanding of risk to support their development of confidence. If this study were replicated, it may be enhanced by a more controlled attempt to also gather pre programme outcome data through questionnaires.

5.6.2.4 Interviews

Two types of interview were used in this RE: semi structured and realist (RI). In any interview there is potential for misinterpretation and demand characteristics, where a participant aims to alter their responses to suit the requirements of the researcher (Robson, 2011). To avoid misinterpretation, both interviews were recorded and participants were given the opportunity to check and refine some of the responses they had given in the semi-structured interview during the RI. The researcher being embedded in research helped to limit misinterpretation through an ethnographic understanding of meaning. In order to avoid demand characteristics, participants were informed at the beginning of both interviews that they were being asked to reply honestly in order to develop a programme specification which focused on how the programme works, rather than whether or not it does. This change in focus seemed to enable participants to answer honestly (including offering comments on hindering or negative mechanisms), rather than inviting a defensive stance.

The semi-structured interviews were conducted individually, but the RI was in a focus group due to the need to save time and enable more natural conversation to flow when discussing the second programme specification (Gray, 2004). Despite the concern that individual’s views can become dominant in a group interview (Cohen, Manion and Morrison, 2011) this did not occur and each participant had the opportunity to

participate in group discussion as well as comment on and rank the programme specification individually (appendix 8.16; 8.17; 8.18). Gaining averages of the ranking assigned to each CMOC meant that the impact of any extreme views were reduced, although there was potential for some bias and skewed results as CMOCs were not ranked by all four participants, due to a need not to overwhelm people with the full data set (Thornbery, 2012).

The final programme specification (appendix 8.19) would be more accurate if parents, pupils and teachers had also been involved in the RI and therefore been asked to comment on the developing programme theory. However, this would have required a great deal of time from these participants who may not have been able to comment on all aspects of the programme, so this idea was rejected in this RE in favour of targeting the Forest School staff who were more able to comment on all areas of the programme specification (Pawson and Tilley, 1997).

5.6.3 Data Analysis

Thematic analysis was used to develop codes and themes from all of the qualitative data gathered and the six steps proposed by Braun and Clarke (2006) were followed carefully in order to support accurate and thorough data analysis. Thematic analysis was selected in this evaluation because of its ability to analysis data inductively and deductively, a feature which was required in this RE where hypotheses from the literature required testing, while remaining open to new emerging themes.

Data analysis would have been improved if more than one researcher had been involved in data gathering, coding the raw data and assigning themes to a group of codes. Although inter-rater reliability was sought to support mutual exclusivity of codes to themes, these participants were not familiar with the complex context of the programme so this may have compromised the accuracy of the process (Fereday, 2006). However, this process was sought to attempt to reduce the subjective impact of a single researcher conducting a full thematic analysis.

It is also acknowledged that there is potential for subjective bias in the way codes and themes were mapped onto the developing programme specification. A single researcher decided whether codes related to features of the context, mechanism or outcome and

there is potential for error here, particular where contextual and mechanism factors are very similar, which can often be the case (Pawson and Tilley, 1997). In order to reduce this impact, each programme specification was reviewed at least three times to ensure duplication hadn’t occurred. Also, the researcher kept in mind Pawson and Tilley’s (1997) advice about what constituted a C, M or O and the final programme specification was checked with critical friends familiar with RE.