Problems with survey implementation and the need fo r follow up

Despite all the precautions taken, several issues arose that highlighted to me potential problems that may occur when conducting research in association with another study, where other people have control over your methods, or when you contract this out to someone else. A few issues arose in working with the HBSC team. For example, I was originally told that schools would be alerted to my follow-up telephone interview which would be sold as part of

HBSC, but this was changed and I had to make the link back to HBSC when I telephoned the school, rather than the link being already established. Also, the team vetoed my preference to put a time limit on when posted teacher questionnaires had to be returned by. They also vetoed my desire to add a unique identifier number to each questionnaire and to record the name of the teacher to whom it was distributed against that identifier. It would have been explained to respondents that this was just to help follow up and that their anonymity was assured with the list of names being destroyed after the project and responses not being linked to individuals. Part way into the study, I raised my concern over not being able to identify survey respondents again, only to find out that my being told that I could not was the result of miscommunication within HPD rather than a methodological decision. Consequently, I phoned the market research company to discuss the possibility of implementing this from that point onwards. They agreed, however only partly as due to a combination of Data Protection and logistical issues (with the former possibly seeming an excuse for the latter), they were only prepared to provide a list of names of staff who had received the questionnaire. Although an improvement, questionnaires could still not be matched to individual teachers, only to schools. This was only one issue of working with the market research company, a relationship that was far more problematic than collaborating with HPD. At the outset, I thought it a fair expectation that asking the teacher to complete the questionnaire during HBSC and collecting at the end would achieve a response rate of nearly 100%. However, Table 5.3 reveals that this was not the case:

1 This may have occurred if HBSC data collection was conducted over two or more visits to a school, or with one year group after another, when the same teacher may have been in charge o f more than one class undertaking the survey.

Table 5.3 Initial teacher survey response rate by number o f returns per school and number o f questionnaires returned

Num ber of responses Num ber of schools Percent of all schools with this num ber of responses Cum ulative % 5 6 10.2 10.2 4 12 20.3 30.5 3 13 22.0 52.6 2 4 6.8 59.4 1 11 18.6 78.0 0 13 22.0 100.0 Total _{59 100 100.0}

O verall response rate as a percentage of all

questionnaires returned (n=136) from all 4 6% questionnaires distributed (n=295)

Note: 61 schools initially took part in HBS but 2 dropped out during data collection, leaving 59. These figures do not include these two schools.

The overall response rate of 46% was much lover than expected, with only 53% o f schools returning 3 or more questionnaires and 22% o f schools returning no questionnaires. While the exclusion of a time limit for postal returns may have lowered the response rate to some extent, most questionnaires should have been collected by fieldworkers. However, there are concerns about the effectiveness of the market research company in implementing this. I had to consistently chase up questionnaires to be returned to me, and on one occasion went to the office to pick up two boxes of outstanding questionnaires. Any contact with the company was hampered by the fact that our contact was acting as an intermediary between myself and those managing data collection. Concerns were also raised over the quality of the fieldwork with regards the teacher survey and there appeared to be no internal quality control over this aspect of the work. Despite fieldworker briefings and the provision of written guidelines, the fieldworker appeared to follow their own procedures. The covering letter was not handed out to teachers with the fieldworker claiming that they never did this unless leaving the questionnaire with the teacher to complete later. Consequently, the potential respondent was not introduced to my project and received no information about why they were being asked to complete the questionnaire,

or instructions on what to do. Also, the fieldworker appeared to make very little effort to collect it back in saying words to the effect that she just leaves it with them and does not know what happens to them after that. Overall, to this fieldworker, and sometimes to the market research company, the teacher survey seemed to be treated poorly, and this appears to have affected the response rate.

Follow up is important to maximise survey response rate (Dillman, 1978) however, due to the low response rate the need 6r this became even more acute. A weakness of Moore et a l’s (2001) study was that with only one questionnaire per school, this raised concerns over the validity and reliability of the data. As such, we felt that it was important to achieve 3 or more responses from each school in order to assess internal inconsistencies in reporting. Consequently, I implemented a thorough follow-up procedure, that was not dissimilar to starting the survey again, in those schools with less than 4 responses. This was hampered however by the fact that, due to the problems described above, in some schools I had a list o f staff who had received the questionnaire and in others I did not. Consequently, follow up took two forms. In schools where the names of all teachers to who the questionnaire had been distributed was provided I sent them a new copy of the questionnaire in the language in which they had received the original, and a covering letter explaining that I was following up non-responses to this questionnaire and asking them to complete the questionnaire and return it if they had not already done so. If they had they were asked to return it uncompleted so that I knew they had completed it previously. These questionnaires all had identifiers, to allow monitoring o f returned questionnaires. The purpose of these identifiers was explained in the covering letter (Appendix 5). In schools with no record of who received the questionnaire, a letter was sent to the HBSC contact, enclosing as many spare copies of the questionnaire as were outstanding from the school (Appendix 6). The covering letter sent alongside this, explained the situation and asked the contact if they could help increase our response rate by distributing the replacement questionnaires to those staff who had not completed their first copy. If the contact did not remember who had originally

received the questionnaire, they were asked to distribute the spare copies to any staff members who reported that they had not received, or completed it, before. An attached information sheet in the form of a flowchart was used to explain this procedure to the contact (Appendix 7).

In this postal follow up, letters were personalised where possible and were all on Cardiff University headed paper and included a real rather than electronic signature in blue ink so that it stood out as handwritten (Dillman, 1978). Freepost envelopes were ag^in used and this time, respondents were also asked to return questionnaires within a set time. These letters were followed up by phone calls. In some cases I managed to speak to the teacher or contact in question. In other cases, messages were taken by administrative staff and distributed to appropriate teachers internally. After many weeks of follow up, the response rate was eventually pushed up to the levels shown in Table 5.4.

Table 5.4 Final teacher survey response rate by number o f returns per school and number o f questionnaires returned

N um ber of responses N um ber of schools Percent of all schools with this num ber of responses Cum ulative % 5 10 16.9 16.9 4 21 35.6 52.5 3 9 15.3 67.8 2 9 15.3 83.1 1 7 11.9 94.9 0 3 5.1 100.0 Total 59 100 100.0

Overall response rate as a percentage of all

questionnaires returned (n=186) from all 63%

questionnaires distributed (n=295)

Note: 61 schools initially took part in HBS but 2 dropped out during data collection, leaving 59. These figures do not include these two schools.

After follow up, only 5% of schools had returned no questionnaires, while 68% had returned 3 or more with an overall response rate of 63%. This exceeds the average 61% response rate for postal surveys (Hox & de Leeuw,

1994 cited in de Vaus, 2002; de Leeuw & Collins, 1997 cited in de Vaus, 2002) although it has been suggested that in surveys of specific homogenous groups such as teachers postal survey response rates may be expected to be much higher than this at above 90% (Dillman, 1978) especially where the topic is relevant to them (Dillman cited in de Vaus, 2002). As well as being hampered by the initial low response which restricted the response rate follow up could achieve, the long length of the questionnaire (Dillman, 1978) as well as the fact that schools are increasingly researched environments may have impacted on this response rate.