Discussion Main findings
Chapter 7 Observations of the practical implementation of screening tests for hearing
in schools
Introduction
Aside from the minimal and very weak evidence for the effectiveness of different implementations of SES reported in the 2007 HTA report,12we were unable to identify any literature on the practical implications
of different implementations or different screen technologies. One paper17evaluated the accuracy of the
HC navigator device in children with a mean age of 6.8 years in schools in the Philippines and reported low sensitivity when used in the school setting where there is significant ambient noise. The use of the PTS has not been formally evaluated in terms of practicality.
Objectives
l To determine the time resource in implementing either of the two alternative screening methods (PTS and HC screener) in primary schools.
l To elicit the views of the school nurses implementing the screening tests.
Methods
This was a prospective observational cohort study. A researcher from the project team observed school nurses while they conducted hearing screening using two methods: the standard PTS and the HC screener. The primary end point was the mean cost per child of implementing each of the two test technologies based on time taken to do each test. Secondary end points included a pass or refer for each test, total time of session, school demographics, nurse opinion on ease of use, how much the nurses would want to use that screener in the future, plus other comments.
The school nurse team was approached by a member of the research team and asked for support with the study. School nurses who delivered hearing screening and were happy to be involved drew up a list of schools in which they routinely screened children for hearing impairment. The research team made contact directly with the head teachers of these identified schools, initially by letter, to gain approval for the researcher to access the school to observe the school nurse. Visits were conducted in all three terms of the school year, in order to cover a range of school conditions and the effects of seasonal infections.
Information sheets were distributed to the school via the school nurses (seeAppendix 6). They were given to parents of children together with information about the school health screen, following the usual process of securing informed consent for the hearing screen. It was explained that, in addition to the routinely used PTS, an extra test (HC) would be carried out by the school nurse and why. Anyone not consenting to either the standard school screen or our extra screen replied to the school nurse,
withdrawing their consent via an‘opt-out’system. Only those giving consent to both tests were included in the study, implied by no reply to the contrary.
Tests during SES were observed in primary schools in the Nottingham area. The number of schools was chosen to ensure resources were measured for at least 180 child screens, representing a range of
catchment populations. Children in the Foundation year or Year 1 (age 4–6 years) were included if they had the usual parental consent for the school screen (following protocols and guidelines for parental consent normally administered by the service) and had not opted out of the SES study.
Data collection took place in the school year from October 2013 to June 2014.
Before the first school session, the researcher met with the nurse to explain and demonstrate how to use the HC screener.
In each school all children>5 years old in Foundation and Year 1 classes were screened by the school nurse. This meant it was necessary for the school nurse and the observing researcher to attend each school more than once. All children in the appropriate classes who did not opt out were screened using both technologies (routine PTS and HC screener), unless the school nurse chose not to perform both tests (for instance because of a lack of attention or nervousness of the child). The order of the tests was randomised according to computer-generated lists provided by members of the research team in Exeter, but if the school nurse felt that the child might have found it difficult to complete two tests, the PTS was performed first to increase the likelihood of completing the routine screening data.
The HC screening method was used as explained inChapter 3. The nurse indicated‘pass’or‘refer’to the researcher. The researcher recorded this, and the time taken for the screen, on the CRF (seeAppendix 6) (note: on the CRF a refer outcome is entered as a fail). Nurses did not use HC data to inform their decision on whether a child should be referred for further assessment by audiology services.
The nurse performed the PTS method according to usual practice. The nurse indicated to the researcher whether or not the child had passed the screen. The researcher recorded the time taken for the screen (not including explanation). Data were recorded on the CRF. If the result of the PTS was to refer the child or it was unclear, the child was rescreened on another day and, if necessary, referred onward as
appropriate according to usual practice.
The start and end time of the session were recorded on the CRF, along with date, school name and postcode, number of pupils on roll, number of pupils eligible for free school meals (a marker of deprivation), and the name of the nurse and the researcher. Comments were also made about the conditions for the screen (e.g. room, noise levels, disturbances, number of children seen and any difficulties during testing). The researcher informed the nurse which screen was to be conducted first for each child according to the randomisation scheme. The total time of the session included time to take children to and from classrooms, break time, screen explanation time, other screens (e.g. vision tests) and interruptions. The CRF included details of what‘other’times were included in the session.
If, during an assessment session, a particular child became upset, uncomfortable, or uncooperative it was up to the school nurse to decide whether or not they continued the session.
Finally, the school nurses were asked on a scale of 0–10 (0 being low) how they would rate each screening test on ease of use, accuracy and how much they would want to use it in the future.
Each participant was allocated a participant number, but to maintain anonymity and to work with the school nurse’s system, no record was kept of which child the number applied to. The original CRFs were kept securely at NHBRU. Photocopies of all CRFs were sent to PenCTU for second data entry and checks. The schools involved received a short summary of the findings at the end of the study.
Sample size
It was anticipated that a sample size of four schools would provide a convenience sample of about 180 children. The study size was not formally calculated.
Analysis
The mean and median time taken to complete the screening tests are presented for: (1) all children, (2) children for whom the PTS was administered first and (3) children for whom the HC screen was administered first. The mean time was compared between the PTS and the HC screener using linear regression models. As the distribution of time to complete the test was skewed, bias-corrected accelerated bootstrap CIs were constructed for the mean difference between the PTS and HC tests.
Results
The three school nurses covering Nottingham East (Carlton, Hucknall, Arnold and Calverton) agreed to do the extra screening. In the catchment areas of the three nurses, seven of the 34 schools covered were willing to take part.
Twenty-two observational sessions were conducted in the seven schools in the Nottingham East area through the school year 2013–14. The parents of four children at the sessions attended did not give consent for the study. Of the children for whom consent was given, 191 were observed, 184 of whom completed both tests; three did not complete either test, the other four did not complete the HC screen. Data were analysed only where the test produced a pass or refer classification. For the remaining participants either the test was not done at all or was incomplete.
Children were seen in groups of between two and five, depending on distance to the classroom and occasionally how disruptive the children were. Sometimes, if the classroom was close by, children were allowed to return to class once they had finished. The total session time was recorded, and included time spent collecting children, administration of other tests (vision) and activities (brushing teeth) and occasionally measurement of height and weight, or retests of previous screens.
Of the 188 PTS tests, 40 (21.3%) were referred. Of the 184 HC tests 71 (38.6%) were referred. Time taken
The mean/median times taken for each screening test were similar for the two tests, at around 1.4 minutes per test, but the range of test times was wider for the PTS (to be expected as the test was not automated). The test time did not appear to vary with the order of the tests. The CI for the mean difference across all screens indicates that the PTS is unlikely to be more than 5 seconds quicker and unlikely to be more than 9 seconds longer on average to administer than the HC screen. That the CI includes zero indicates that it is plausible that there is no difference between the PTS and HC screener in mean time taken (Table 25).
TABLE 25 Time taken (minutes) to do screening tests and comparison between the PTS and HC screener
Test n Mean (SD) Median (IQR) Range Mean difference (PTS–HC)
Across all occasions
PTS 188 1.39 (0.67) 1.24 (1.05–1.55) 0.63–7.5 0.002 95% CI–0.08 to 0.14a
HC 184 1.39 (0.24) 1.33 (1.27–1.58) 1.03–3.47 When the PTS is administered first
PTS 105 1.37 (0.42) 1.28 (1.08–1.57) 0.73–3.22 –0.007 95% CI–0.08 to 0.09a
HC 102 1.38 (0.18) 1.33 (1.28–1.45) 1.12–2.45 When the HC is administered first
PTS 83 1.42 (0.89) 1.2 (1.03–1.50) 0.63–7.5 0.013 95% CI–0.13 to 0.33a
HC 82 1.41 (0.30) 1.33 (1.25–1.45) 1.03–3.47
Observations of researchers and nurses
The researchers observed that schools were often unsuitable for testing hearing because they were too noisy and a suitable alternative room was often not available. On some occasions the nurse had to give up with the hearing tests and return on a different day.
The nurses suggested advantages and disadvantages of each test (Table 26)
The three nurses scored each of the tests on a total of 20 occasions for 185 children (no tests at one session because no suitable room available; hearing testing abandoned at another session, after three children tested, owing to noise). All nurses scored all tests as 5 or above for ease of use, accuracy and future use. The mean, median and range of scores are shown inTable 27. The PTS test scores higher than the HC test on all measures but in terms of ease of use that might have been because the nurses were
TABLE 26 Advantages and disadvantages of the PTS and the HC screener as observed when used by school nurses
PTS HC
Advantages
Provides a full audiogram if needed Lightweight Can turn it up in noisy situations Portable Can turn it up to check a child’s
understanding
Hygienic The headphones help block background
noise
No need for mains electrical socket No need for headphones
Disadvantages
Headphones can be tight Cannot pause to check understanding
Headphones disliked by some children Cannot pause to wait for background noise to stop
Needs to be plugged into a socket Cannot vary timing of presentation so some children anticipate when to put up their hand (particularly if they see their peers doing the test before them) Cannot repeat a particular tone
Only plays six tones–and only two frequencies are tested
The tone at 20 dB is very quiet in a school situation and a lot of children miss it Younger children found this more difficult to understand
No time between tones to give praise or encouragement The cups can fall off
The equipment can get in the way of the child’s hand going up
TABLE 27 Ratings of three nurses on the practical implementation of the PTS and HC screener in schools on 20 occasions–scoring from 0 (low) to 10 (maximum)
Attribute
PTS HC
Mean Median Range Mean Median Range
Ease of use 8.75 9 7–10 8.40 8 6–10
Accuracy 8.70 9 7–10 6.45 6 6–9
more familiar with using the PTS. They also rated accuracy more highly for the PTS than the HC screener. When asked how much they would want to use a test in the future they again rated the PTS higher, but commented that they could see the HC screener as being useful as a back-up in some situations.
For two of the nurses the scores all either decreased or stayed the same over time. For the third nurse the HC screener scores stayed the same and the PTS scores increased for ease of use and future use and decreased for accuracy over time.
Discussion
In order to assess the practical issues associated with using the PTS or HC screener as the screening tests in a SES programme, careful observation of screening 184 children in seven schools in the Nottingham SES programme was undertaken. Three nurses covered 22 sessions. Each child had both tests applied, with the test given first being randomly determined.
The average time taken to implement each screening test was nearly identical at about 1.4 minutes, noting that the variability in time was greater for the PTS than for the HC screener. The difference in average time taken to conduct the test was not statistically significant and this was not affected by which test was the first to be used.
Nurses slightly preferred the PTS, but acknowledged that the HC screener could still prove a useful backup for children who refused to wear headphones or when there was no electrical socket available to run the PTS test. The observations about potentially noisy school environments raise the possibility that the accuracy of screening tests may be overestimated in the quieter research environment experienced in diagnostic accuracy studies, including our own.
Strengths and weaknesses
The study was prospective and undertaken in accordance with a protocol without deviation. The target number of participants was achieved. The observations were carried out in familiar
environments for the children allowing assessment of the operation of the screening tests without being compromised by the children feeling anxious–an issue that may have hindered the testing of control children at NHBRU in the diagnostic accuracy study.
Testing took place in a number of different schools throughout the school year, enabling the capture of seasonal changes that might affect hearing through colds, illness or hay fever.
The observations captured data on a range of children from different backgrounds. Several city schools were observed, enabling examination of how the hearing screens by school nurses might be affected by school size, behavioural challenges, support from teaching staff and school facilities, such as test room conditions.
We note the limitation that feedback on the two screening tests and the observations of testing involved only three nurses. However, all nurses undergo the same training and follow a set protocol while screening and therefore procedural differences should not affect test performance. With regard to verbal feedback provided by the nurses on the two tests, all three demonstrated inter-rater reliability, finding similar strengths and weaknesses for both the PTS and HC tests. Gaining opinion from a wider nursing community is unlikely to affect the conclusions drawn.
Results in the context of other studies
As far as we are aware this is the first study to systematically examine the practical issues associated with applying tests that might be used in SES with children in real-life circumstances.