Pilot Study Design: Strengths and Limitations

Methods were piloted, with a view to identifying study design requirements for future research on New Zealand children’s sleep. A brief evaluation of the strengths and limitations of recruitment procedures and measures is outlined.

4.6.1 Recruitment

Information about the pilot study was primarily distributed via printed flyers that were sent home with children from school. Using schools as the predominant recruitment sites enabled the targeting of children in Year 3, thereby minimising developmental differences in the sample. Schools with specific decile ratings were also able to be targeted. However, the aim of recruiting half of the sample from low decile schools was not met. This may have been due, at least in part, to the reliance on parents reading the flyers and having to contact the researcher by phone or email. Language may have been a barrier for families with English as a second language, some families may not have had phones or email access, and others may not have felt comfortable contacting a researcher they had not previously met.

Forty six schools agreed to flyer distribution, resulting in 83% of the sample being recruited. In comparison, one day at the shopping mall providing face-to-face and take home printed information, resulted in 7% of the sample participating. This indicates that a more personal approach may improve recruitment rates in the future. This could include contact with families via community groups, marae, churches, sports clubs, markets and local events. A replication of the shopping mall information day is recommended, as well as making contact with families via schools, but using a more personal approach. Speaking to children at assemblies and in class time may provide children with a better idea about the study. Being available to speak to parents at school fairs and parent/teacher evenings may also improve recruitment rates of a more diverse group of families.

4.6.2

Measures

Feedback received from parents indicated that the actigraph was not difficult to use. The actigraphs were lightweight and able to be worn by all participants without

DISCUSSION

130 any known adverse affects. One family recommended the use of Velcro straps in the future, to enable a better fit on children’s wrists. An alternative may be sourcing paediatric wrist bands for the AW-2™s.

Some confusion was apparent regarding use of the event marker, therefore clearer written and verbal instructions should be provided in the future. The advantage of using the actigraphs was that children could manage them independently, including pushing the event marker and taking the Actiwatch™ off as required, such as when swimming. The limitation of this was demonstrated when two actigraphs were temporarily misplaced (but relocated).

Four families were asked to repeat one week of recording. One was due to the wristband buckle breaking and having to be replaced, and the others due to technical problems with delayed recording not commencing. It is therefore recommended that watches be activated immediately prior to distribution to families in the future, to minimise the chance of having to repeat data collection.

Compared with Acebo et al. (1999) who reported that up to 28% of actigraphic recordings may be unusable, 92.9% of night time data were retained and 80.8% of daytime data were usable. This indicates that the protocols for data collection, screening and scoring were effective. However, Acebo and colleagues (1999) found that data aggregated over one or two nights showed poor reliability for sleep measures. As it is important to measure children’s sleep on school and non-school nights in order to identify differences across the week, it is recommended that a future study incorporates at least 10 consecutive days and nights of recording to provide two weekends of data. A standardised approach to the starting day and time of recording may also be beneficial, to minimise confusion for families as well as to have more consistent periods of data (for example recording from 08:00 Friday for 10 consecutive days and nights).

Actigraphy data were analysed using a ‘medium’ sensitivity threshold. This may have resulted in conservative estimates of sleep, as Hyde et al. (2007) concluded that using the ‘low’ and ‘auto’ sensitivity thresholds were best at predicting sleep in children and ‘high’ and ‘medium’ settings were better for predicting waking. Notably, their study sample consisted of 45 children, who were being seen for potential sleep disordered breathing, and therefore may not have been representative of the normal population.

131 Guidelines for manually screening the actigraphy data were used by two independent researchers, who both scored a sample of 11 files (21.2%). An overall agreement of 93.9% indicated that the rules for identifying rest and excluded intervals enabled consistent processing of data. It is recommended that a minimum of 20% of files be double scored in future study protocols.

Due to the limitations of actigraphy outlined in Section 4.5.5, results of sleep quality, and sleep latency, analyses must be interpreted with caution. The potential incorporation of home-based PSG with a sub-sample of children in the future could therefore provide useful comparative data. It is recommended that this take place over three nights, to incorporate one adaptation night, one school night, and one non-school night.

Child’s Sleep/Wake Diary

Diary data were double entered into SPSS 16.0 databases by two researchers, with overall agreement of 98.4%. It is recommended that a future study protocol incorporates this process prior to analyses, to ensure accurate data entry. The majority (76%) of parents found the diary easy to use and one (2%) found it difficult. As per parental feedback, it is recommended that the timeline be amended so that it does not use the 24 hour clock, and that more space is provided for comments. Adding childcare and parental work patterns to the diary would enable more accurate estimates of family routines and their association with children’s sleep. Caffeine consumption, technology use, and exercise, particularly within two hours of bedtime, would also be beneficial to incorporate into the diary.

Notably, parents reported confusion regarding the diary questions on children’s ‘quiet’, ‘moderate’ and ‘physical’ play. As this raised concerns about how accurate parental estimations were, the decision was made to not use parentally reported activity levels and duration data for further analyses. It is therefore recommended that a different method of gathering subjective waking activity data be explored. One alternative is the multimedia activity recall (MARCA). This is a computerised time use diary, which has been found to be a valid self-report measure of activity in 9 – 16 year olds (Olds, Ridley, Dollman, & Maher, 2010).

DISCUSSION

132

Child/Family Questionnaire

All families who provided feedback found the questionnaire to be ‘easy’ or ‘OK’ to use. Questionnaires were double entered into SPSS 16.0 databases, with overall agreement of 94.2%. It is recommended that questionnaire data be double entered in future studies to ensure accuracy of data entry. As already discussed, childcare timing and duration would be more accurately captured in the diary, as opposed to the questionnaire, therefore this section should be removed from the current questionnaire format. Analysis of children’s developmental level may be added in future versions, following paediatric consultation regarding appropriate measures. Factors to consider including in the future include children’s mood, behaviour (such as hyperactivity), emotional lability, and academic performance, as well as parental sleep patterns, mood, and parenting style. However, it is important to balance the amount of information being gathered against the burden placed on participating families, and the impact on compliance and data accuracy.

Children’s Sleep Habits Questionnaire (CSHQ)

As outlined in Section 4.4, CSHQ scores in the current sample were comparable to those reported by Owens et al. (2000). The cut-off total score of 41 provided a useful measure of parentally reported sleep problems. Amendments made to the CSHQ for the pilot study enabled subjective reports of school and non-school sleep/wake behaviour, as well as parental perception of problem sleep, to be measured. It is recommended that these adaptations be retained for future studies. As per parental feedback, it is also recommended that a column labelled ‘never’ be added as an option to questions on sleep behaviours. Revalidation of the amended format would therefore need to be carried out. In general, the CSHQ was an effective tool for subjectively measuring the sleep/wake behaviour of children in the sample, and was completed by parents with no apparent difficulty.

4.6.3 Procedures

Parental feedback indicated that, on the whole, adequate information was provided before, during and after participation in the pilot study. The exception was information about the use of the event marker, as already discussed.

Limitations were apparent in the screening process. Although each family was asked verbally whether their child was in Year 3 at school, as well as printed

133 information including this, two children’s data could not be used as they were in Year 2 and Year 4. One factor that may have contributed to this was that the initial flyer did not include the wording “and in Year 3 at school”. This was amended in the early stage of the pilot. Secondly, although schools were asked to distribute flyers only to Year 3 pupils, one school sent flyers home to all children in a composite Year 2/3 class.

Parents were asked to delay actigraph and diary use if children were sick, however one set of actigraphic data could not be used in analyses due to the child being sick and taking medication on all seven days of recording. This needs to be clarified in the written information provided to parents, as well as reiterated verbally in the future.

Feedback from parents was positive regarding the follow up procedures in the protocol. It is recommended that a mid-week phone call or email to parents be retained in future study design. Prompt provision of children’s actograms was also appreciated by parents, as was the certificate, pen and stickers sent to children. It is important, therefore, to incorporate feedback and acknowledgement in future protocols. Overall, few problems were encountered and compliance levels of families were high, indicating an effective study protocol.