Usual alcohol-related activities in both intervention and control schools over the course of the study
As part of the online teacher survey (see Online survey with teachers), intervention and control teachers were asked to provide details of all prevention and health education that had been delivered in the school (i.e. not limited to year groups receiving STAMPP) in addition to regular PD curricula. The assessment took
the form of a self-completed monthly‘diary’, and teachers were asked to include any activities that they
thought were relevant. These were retrospectively classified by the research team. This was undertaken in order to assess the provision of competing or additional alcohol education in the trial sites.
The SHAHRP intervention was delivered in most schools as part of their personal, social, health and economic (PSHE) (or local equivalent) provision, and did not replace statutory activities. Overall, 88.5% of intervention schools self-reported at least one activity that they thought was relevant, resulting in a total of 58 discrete monthly activities. However, of these, very few were classed as relevant to alcohol prevention. Two intervention schools reported provision of eight separate 1-hour-long alcohol information sessions delivered by a local, non-governmental organisation across the time period, and one of these sessions was delivered to pupils who would have also received the SHAHRP. One respondent reported a single 1-hour
information session on alcohol delivered by the local police force, but this was to pupils who were older than those in the trial sample. One school reported two presentations of a theatre-based alcohol
intervention but did not specify the target age group. Finally, one school reported a generic 1-hour session on harms in the family home, which included a discussion of alcohol.
In 93.3% of control schools, participants reported at least one activity that they thought was relevant, resulting in a total of 47 discrete monthly activities. Of these activities, two respondents reported generic alcohol awareness sessions across different year groups, including pupils in the target years, and one respondent reported a single 1-hour information session on alcohol delivered by the local police force. Based on these reported data, it was concluded that provision of additional alcohol education/prevention activities in control and intervention sites was minimal (see also Chapter 6, Self-assessment of risk of bias). Fidelity of implementation of STAMPP
Methodology
Intervention teachers were asked to complete two self-report surveys concerning fidelity and completeness of delivery of the two phases of SHAHRP. Although a method such as independent observation may
produce more reliable and valid data,90–92it was not possible to utilise this method because of time and
resource limitations. Self-report surveys were therefore considered the most efficient form of data collection because a large number of teachers were involved in delivery of the programme.
The surveys used to self-assess fidelity are included in Appendix 7. The respondents were required to indicate the extent to which they delivered the activities within each phase. They could choose from three response options: 0,‘not at all’; 1, ‘partially’; and 2, ‘fully’. The participants were also asked to indicate how many lessons in total they needed to complete the activities. To be delivered as intended, phase 1 should have been completed in six lessons and phase 2 in four. Finally, the participants were asked to indicate the degree to which they used the accompanying CD to support project delivery (a CD accompanied both phases 1 and 2). They could provide their answer using a 10-point Likert response option: 1,‘I never used it’, to 10, ‘I used it at all times’.
After delivery of each phase, the intervention teachers were sent an e-mail that contained the relevant self-report survey and an invitation to complete it. If more than one teacher within a school delivered SHAHRP, the aforementioned e-mail was sent to the contact teacher of the school, who was asked to distribute the questionnaire to the teachers who had delivered the programme. Each questionnaire took
2–3 minutes to complete. The data collected were exported to IBM SPSS version 21.0 for analysis.
Phases 1 and 2 contained 17 and 12 activities, respectively. If participants indicated that they did not complete an activity at all, this activity was assigned a score of zero. If they indicated that they partly or fully completed an activity, this activity was assigned a score of one. The scores from each activity were summed. The activities in phases 1 and 2 should be completed in six and four lessons, respectively. In order to reflect this requirement, the summed activity score was divided by the number of lessons that the teacher
indicated was needed to complete the material.
However, it was also possible that the activities were completed in a fewer lessons than required (suggesting that not all material had been covered adequately). The summed activity score could not be divided by the number of lessons in this case, as it would produce an inflated score. In this scenario, the number of lessons needed to complete the material was recoded to reflect the distance from the ideal number. For example, if a teacher indicated that the activities were completed in five lessons, one below the ideal number in phase 1, the number of lessons was recoded as 7, one above the ideal number. The summed activity score was then divided by this recoded number.
PROCESS EVALUATION: METHODS AND RESULTS
NIHR Journals Library www.journalslibrary.nihr.ac.uk
The use of the CD to support project delivery accounted for the final 10% of fidelity of implementation of
each phase. The Likert response 1,‘I never used it’, was recoded to 0 in order to represent 0% fidelity in
terms of the use. The other ratings remained the same. For example, a score of 5 accounted for 5% of overall fidelity of the phase.
The two scores were added to provide a composite indicator of fidelity of implementation for the classroom intervention for each phase.
Owing to resource constraints, less information was collected on the delivery of the parental intervention. Data are presented on attendance at the parental evenings and the number of questionnaires returned, which were used as an indicator of acknowledgement of reading the mailed information.
Results
School intervention
Overall, across all intervention schools that returned data (47; 87%), phase 1 was assessed to have been implemented with 72.69% fidelity and phase 2 was implemented with 68.76% fidelity (Table 22).
There was variation between schools with respect to the number of lessons in which content was delivered. The mean number of lessons needed to complete all the activities in phase 1 was 8.09 (SD 2.61), whereas activities should have been completed in six lessons (Table 23). The mean number of lessons needed to complete the activities in phase 2 (Table 24) was 6.65 (SD 3.02), whereas activities should have been completed in four lessons. The CD of supporting materials was used with 74.8% fidelity in phase 1 and 69.8% fidelity in phase 2.
Table 38 (see Appendix 8) presents fidelity scores for each school that returned data. These scores ranged from 28.23% to 94.35% in phase 1 and from 34.71% to 100% in phase 2.
TABLE 23 Components of generated phase 1 fidelity of implementation score
Component Score (%) Mean (SD) Minimum possible Maximum possible Minimum recoded Maximum recoded Summation of activities 15.92 (2.18) 0 17 0 17
Mean number of lessons 8.09 (2.61) 0 N/A 3 20
Recoded lessons 8.32 (2.40) 6 N/A 6 20
Activities/lessons 2.06 (0.54) 0 2.83 0 2.83
CD 7.48 (2.76) 0 10 0 10
N/A, not applicable.
TABLE 22 Overall fidelity of implementation scores
Phase of delivery
Score (%)
Mean (SD) Minimum Maximum
Phase 1 72.69 (17.98) 28.23 94.35
TABLE 24 Components of generated phase 2 fidelity of implementation scores Component Score (%) Mean (SD) Minimum possible Maximum possible Minimum recoded Maximum recoded Summation of activities 12 (0.00) 0 12 12 12
Mean number of lessons 6.65 (3.02) 0 N/A 3 20
Recoded lessons 6.72 (2.97) 4 N/A 4 20
Activities/lessons 2.06 (0.70) 0 3 0.60 3
CD 6.98 (2.65) 0 10 0 10
N/A, not applicable.
Parental intervention
Overall, parental/carer participation was low. In total, 319 parents/carers attended the intervention nights in NI (9% of those eligible), and 63 parents attended in Scotland (2.5% of those eligible).
With respect to the mailed intervention, 1074 returns were received from parents/carers in NI (31% return rate) and 440 were received in Scotland (18% return rate).