vi Reliability and validity issues associated with the questionnaire

Robson (2002) points out that the reliability and validity of survey data depend to a considerable extent on the technical proficiency of those running the survey. He suggests that internal validity issues can arise if the questions are incomprehensible or ambiguous, because valid information is not being obtained. The researcher tried to reduce the possibility of misunderstanding instructions and questions in the survey by piloting the questionnaire with the APCs and with her university tutor and using their feedback to adjust the design of the questionnaire accordingly. It can be seen from Table 3.6 and Table 3.7 that some of the questions in the questionnaire could have been worded in a way that may have tapped better into the research questions being investigated. For example, Question 3 about the most ‘useful aspects’ of the project team’s work could have been

worded in a more specific way for each of the research questions: for instance, ‘What was it about the training that built staff’s confidence and skills in using this approach?’ This would have revealed mechanisms around an assumption that the training had built their confidence and skills and would have produced more robust answers to research question 3. An alternative way of finding out about the impact of training would have been to observe staff delivering the intervention.

Robson (2002) also emphasises that securing involvement from the respondents to a questionnaire can be difficult. In the case of the Treasure Project evaluation, the researcher was aware that head teachers’ responses and willingness to complete the questionnaire may be influenced by their views about the Treasure Project. They might want to please the researcher through their responses to the questionnaire, or might view it as a way of drawing in further resources or funding. It should be noted that although the Treasure Project no longer existed as a funded project, the work of the team continued as a traded service from which schools could commission packages of support. The move towards income generation may have resulted in some feelings of abandonment, particularly for the two school clusters that had been targeted initially, and that had been able to avail

themselves of a wide variety of ‘cost-free’ support from the project team, including funding for staff to attend training courses. In order to pre-empt any misunderstandings of the purpose of the questionnaire the researcher tried to design the questionnaire so that head teachers would be aware of its purpose before beginning to answer any of the questions. The accompanying letter (see Appendix III) also stated the purpose of the questionnaire and the use of Educational Psychology Service headed paper for the letter would also reassure head teachers of the intention and genuineness of the enquiry. In this

way it was hoped to redress any external validity issues arising from understanding of the purpose of the evaluation. Further external validity issues might be due to head teachers not necessarily being completely neutral in their views about their own school, their school staff, or the impact of the interventions within their schools. This might also be true of other stakeholders’ opinions. The involvement of the APCs by the researcher would help to triangulate some of the general interpretations of the data. Some of the head teacher

questionnaire data from ‘successful’ schools would also be triangulated to some extent through the second phase of data collection.

The willingness of the recipients to answer questionnaires may affect the validity of the data collected (Robson, 2002). It should be noted that, for this case study, the original Treasure Project schools had not volunteered to take part in the mental health and emotional wellbeing project, but had been chosen by the Children’s Fund, CAMHS and other key stakeholders. Some of these schools were keener than others to be involved in a multi-level participatory partnership to promote the mental health and emotional wellbeing of children within their school community. A few of these schools wished the project team to only work directly with children presenting with emotional and behavioural issues and were reported by the project team as being resistant to a more whole school approach. Likewise, schools outside the initial two clusters that requested support from the Treasure Project did not necessarily wish to consider a more multi-level model to the promotion of emotional wellbeing. In addition, both Project and Non-Project Schools received the same questionnaire, which asked about the impact and outcomes of the project. This may also have caused some confusion for the Non-Project Schools as they may not have considered themselves as being involved with the Treasure Project at all. Therefore some of the

questions in the questionnaire may not have been relevant to those head teachers’ experience or understanding of the Treasure Project. The researcher took the decision to send the same questionnaire to all schools in order to increase the external validity. As the head teachers’ attitude towards involvement with the Treasure Project was not explored this may, however, have decreased the internal validity of the some data gathered through the questionnaire.

One of the reliability issues that might arise from the chosen data collection tool being a self-completed questionnaire is that the respondent can answer the questions in any order, and this may affect the answers given (Robson, 2002). He suggests that a high reliability of response to a questionnaire can be ensured by presenting all respondents with the same standardised questions, carefully worded after piloting. As previously mentioned, the questionnaire was piloted with the two APCs and the researcher’s university tutor. A self- completed questionnaire also allows the respondent to confer with others in order to answer the questions; this may mean that the head teacher’s comment may in fact be the Special Needs Coordinator’s views. To decrease these reliability issues the researcher tried to ensure the response rate was as high as possible so that information would be received from as many practitioners as possible. The researcher also thought that the head teachers would be more likely to complete the questionnaire if it was not too lengthy and was simple in format.