Evaluation of the Computer-Assisted Speech
Perception Assessment Test (CASPA)
Carol L. Mackersie* Arthur Boothroyd* Donna Minniear*
Abstract
Interlist equivalency and short-term practice effects were evaluated for the recorded stimuli of the Computer-Assisted Speech Perception Assessment (CASPA) Test. Twenty lists, each consisting of 10 consonant-vowel-consonant words, were administered to 20 adults with normal hearing . The lists were presented at 50 dB SPL (Leq) in the presence of spectrally matched steady-state noise (55 dB SPL Leq) . Phoneme recognition scores for the first list presented were significantly lower than for the second through the twentieth list presented, indicating a small practice effect. Phoneme scores for 4 of the lists (3, 6, 7, and 16) were sig-nificantly higher than scores for the remaining 16 lists by approximately 10 percentage points. Eliminating the effects of interlist differences reduced the 95 percent confidence interval of a test score based on a single list from 18.4 to 16.1 percentage points. Although interlist dif-ferences have only a small effect on confidence limits, some clinicians may wish to eliminate them by excluding lists 3, 6, 7, and 16 from the test. The practice effect observed here can be eliminated by administering one 10-word practice list before beginning the test.
Key Words: Computer-Assisted Speech Perception Assessment (CASPA), list equivalency, outcome assessment, phoneme recognition, speech audiometry, speech perception, word recognition
Abbreviations: ANOVA = analysis of variance, CASPA = Computer-Assisted Speech Perception Assessment, CASPA = Computer-Assisted Speech Recognition Assessment, CVC = consonant-vowel-consonant
S
peech recognition measures are widely used in audiologic practice for diagnostic assessment but are not widely used in hearing aid outcome assessment . Several limi-tations of performance measures have led to their decline in popularity as a performance component of a hearing aid outcome battery. First, it is widely recognized that conventional speech recognition measures have poor test-retest reliability (Shore et al, 1960 ; Boothroyd, 1968a; Thornton and Raffin, 1978). Poor reliability compromises test sensitivity, making it difficult to detect differences between speech recognition scores under different lis-tening conditions . Conventional speech recog-nition measures are also time consuming to*Department of Communicative Disorders, San Diego State University, San Diego, California
Reprint requests : Carol L. Mackersie, Department of Communicative Disorders, San Diego State University, 5500 Campanile Drive, San Diego, CA 92182
administer and score. Many audiologists argue, therefore, that performance measures of hear-ing aid benefit are not a realistic option for a typ-ical audiology practice.
There are, however, several advantages to including performance measures in a hearing aid outcome battery. First, performance mea-sures can provide objective documentation of speech recognition benefit attributable to hear-ing aid use. In addition, performance outcome measures can supplement subjective outcome assessments because they tap different and independent aspects of function (e.g., Bentler et al, 1993a, b).
There are several considerations in choosing a test protocol for performance outcome assess-ment. First, to sample a range of listening situ-ations important to the hearing aid user, listeners should be tested at several input levels and in both quiet and noise. Multilevel testing is par-ticularly valuable in the evaluation of the effects of compression on speech recognition. Second, there are advantages to including both word and
sentence materials in the assessment . The use of word materials allows a valid measure of phonemic perception while minimizing the effects of language knowledge and the use of context (Boothroyd, 1968b; Boothroyd and Nittrouer, 1988). The use of word materials also permits a detailed analysis of phoneme errors. Sentence materials, on the other hand, are more repre-sentative of speech encountered in the real world and allow for such factors as the influence of suprasegmental features and short-term mem-ory. It is possible, however, to predict sentence recognition from both word and phoneme recog-nition scores (Boothroyd and Nittrouer, 1988). Finally, the reliability, sensitivity, and efficiency of the test must be considered as factors crucial to demonstrating statistically significant differ-ences between speech recognition scores for dif-ferent conditions within the time constraints of clinical practice.
The reliability of' speech recognition tests can be improved by increasing the number of test items. Phoneme scoring increases the number of items in a word recognition test without requiring additional test time (Boothroyd 1968a;
Olsen et al, 1997; Gelfand 1998) . The use of phoneme scoring triples the number of test items in a consonant-vowel-consonant (CVC) word list because three items are scored for each word
rather than just one. I The increase in the num-ber of storable test items reduces (improves) the confidence limits of the test. This means that smaller differences between listening con-ditions can be detected (Boothroyd, 1968a; Thornton and Raf'fin, 1978).
Until recently, phoneme scoring has been regarded as awkward to implement because of extra time required to score the individual phonemes . Two recently developed computer-assisted word recognition tests, however, provide the option to score individual phonemes easily and quickly. Gelfand (1998) developed the Computer-Assisted Speech Recognition Assess-ment (CASRA), a computer-administered test that was designed to provide 450 storable items in 50 three-word trials . At each trial, the subject is asked to repeat three CVC words . The tester then indicates on the computer keyboard which of the nine phonemes are correct . By using three words in each trial rather than one, the number
of test items in a 50-trial test increases to 150 if whole-word scoring is used and to 450 if phoneme scoring is used. Each 50-trial test takes approximately 6 minutes to administer. Unfor-tunately, at the time of writing, this test was not expected to be commercially available in the foreseeable future (S Gelfand, personal com-munication, 2000).
Another computer-assisted test that is cur-rently in use by both researchers and clinicians is the Computer-Assisted Speech Perception Assess-ment Test (CASPA) (Boothroyd,1999) . The CASPA test, which was originally designed to assess hear-ing aid outcome, permits multilevel testhear-ing in quiet and in noise. The interactive software inter-faces with the tester to allow computer presenta-tion and scoring of the CVC stimuli and provides separate scores for words, phonemes, consonants, and vowels. Like the CASPA test described above, the phoneme scoring feature of CASPA increases the number of test items to improve the confi-dence intervals of the test scores without addi-tional administration time. It takes approximately 1 minute to administer one 10-word list, which yields a score based on 30 phonemes. By admin-istering two to three lists under each condition, one can substantially improve the likelihood of detect-ing significant differences between phoneme recog-nition scores measured under different listening conditions. Moreover, multilevel testing with the CASPA test is faster than multilevel testing with a conventional word recognition test because of CASPA's ease of administration and automatic phoneme scoring .
Although the CASPA test is increasing in popularity among researchers and clinicians, there is currently no information available on the list equivalency for the recorded stimuli or on practice effects for these materials . The pur-pose of this study was to gather this information.
METHOD Participants
Twenty normal-hearing adults served as listeners. Ages ranged from 19 to 40 years, with a mean of 23.4 years. All listeners had pure-tone thresholds of 20 dB HL or less for octave fre-quencies between 250 and 8000 Hz
CASPA Stimuli
'Lexical redundancy actually reduces the number of statistically independent items from 3 to approximately 2.5 for listeners with normal language skills (Boothroyd and Nittrouer, 1988)
The CASPA stimuli consist of 20 lists of CVC words with 10 words in each list. The lists are isophonemic, that is, each list contains one
example of each of the same 30 phonemes . The 20 CASPA lists include the original 15 AB iso-phonemic word lists (Boothroyd, 1968a, b) and 5 new isophonemic lists (Oruganti, 2000). The stimuli available with the CASPA test at the time of writing are digitized recordings of these 20 lists spoken by a female talker. The noise used in the test is steady state and is spectrally matched to the female talker's long-term average spectrum. When testing in noise, the speech items are temporally centered in the noise, the duration of which is fixed at 2 seconds. The word lists are provided in the Appendix.
CASPA Software
The software controls the presentation of the CASPA words and provides the option to present stimuli in quiet or in the presence of noise. The noise and speech can be delivered through sep-arate channels (stereo) or mixed in a single channel (mono) . The program can be used to provide speech input to an audiometer, an ampli-fied loudspeaker, or another external device. The presentation level can be controlled either through the CASPA software or with the atten-uator of an audiometer . If the CASPA software is used, the speech level is selectable in 5-dB steps between 45 and 75 dB SPL (Leq), and the noise level is fixed at 55 dB SPL. If an audiome-ter attenuator is used to control the presenta-tion levels, then a wider range of intensities can be selected, and both the speech and noise levels can be adjusted. The software permits randomization of the order of presentation of the lists and of the items within the lists.
Figure 1 shows an example of a test screen as it appears to the tester during administration. Following presentation of a test item, the tester types the listener's response into the computer. The computer then displays both the test item and the listener's response. Using the computer mouse, the tester selects the box(es) corre-sponding to the phonemes repeated correctly. In the particular example shown in Figure 1, the test item was "both" and the listener's response was "boat," so the tester marked the first con-sonant and the vowel correct. For testers who are not comfortable with the mouse, responses can be scored from the keyboard using either "1," "2," "3" or "C," "V" "B" to score the first, second, and third phonemes, respectively. The option also exists to postpone scoring until after the subject has left.
For each list, the software tabulates the total number of words, phonemes, consonants,
and vowels repeated correctly (see Fig. l, lower left). In addition, both the stimuli and corre-sponding subject responses are logged on the computer's hard drive for each test item. At the end of a list or series of tests, the results can be printed through the software in graphic (performance-intensity functions) or tabular for-mat. Alternatively, the results can be read into a spreadsheet program for error analysis. Procedures
The stimuli were played through a Creative-Labs SB-16 Sound Card at the rate of 22.05 k/sec with 16-bit resolution. The output of the sound card was routed to the tape input of a Gra-son-Stadler GSI-16 audiometer and delivered monaurally to TDH-59 earphones. Speech stim-uli were presented at 50 dB SPL (Leq) in the presence of the spectrally matched noise (55 dB SPL). A signal-to-noise ratio of -5 dB was used to avoid ceiling- and floor-level performance. Avoiding ceiling- and floor-level performance
increased the likelihood of detecting interlist differences. In the interest of efficiency, no car-rier phrase was used. Listeners were instructed to repeat the words and were encouraged to guess. The tester scored the listeners'responses using the computer software described above.
The presentation order of the 20 lists was counterbalanced among listeners using a Latin-square design. The order of items within each list was randomized each time the list was pre-sented. Testing was completed in one 40-minute session. Data were transformed to rationalized arcsine units prior to statistical analyses to sta-bilize the error variance (Studebaker, 1985).
RESULTS
Order Effects
Group mean phoneme recognition is shown in Figure 2 plotted as a function of presentation order. Note that the score for the first list pre-sented is lower than the scores for the subse-quent lists. The solid line in the figure is a least-squares fit to an exponential growth func-tion. The formula for this function is
Y = a + (b - a)* (1- ec-cx-i)i~>)
where a = intercept (score on trial 1), b = asymp-tote, and c = time constant (i.e ., the number of lists required for the gap between the current score and the asymptote to be reduced by approx-
Subject ID (max. 8 characters) S1 Type subject's response and hit Enter If the whole word was correct just hit Enter.
Carrier
None
If the subject gave no response type 0 (zero) or- .[
boat
Speech level in dBSPL (leq) 50 The stimulus word was Acoustics [Noise SpeechiNoise mono J
o ~
th
Listening conditions for the two ears:Left J Right Earph-~ Check correct sounds below and click OK
Vowel
List # J
Start over ~+ll
Current word # Consonant 1 FV r r (B)Consonant 2 None (- All 3
d ?
Judge Loudness? F- Randomize wor s loo ear graph Q t N i u e i o se
- h
Postpone scoring? r Randomize lists? p Aided:
50 Other : o o Words Phonemes 2 C 1 v c ons c % unaided: o Normal: - --- 0 0 2 0 4 0 6 0 8 0 1 00d BSP L(leq) R
Return Preferences 0 Instruction MatrixTesting Phoneme scoring BSI ~J Data retrieval °°
Figure l CASPA test screen showing test parameters and sample trial for one subject. The test stimulus ("both") is shown in the second section at the right, the listener response ("boat") is shown at the top right, and the score for the test item is shown in the middle-right section.
imately two-thirds) . Note that the nonlinear correlation coefficient reflects a significant increase in phoneme recognition scores as the number of lists presented increased (r [171 = .56, p < .01). The mean score rises to within 3 percentage points (one phoneme) of the maxi-mum after 0.7 lists . That is, by the second list presented, the mean score is close to the asymp-
tote. A one-way repeated-measures analysis of variance (ANOVA) confirms that the significant effect of presentation order (F = 6.69, df = 19, 361, p < .0001) disappears when the first list pre-sented is excluded from the analysis (F = 1.30, of = 18, 342, p > .05). Taken together, the non-linear regression and ANOVAs indicate the pres-ence of a small practice effect, but performance is stabilized by the time the second list is presented.
1 2 3 4 5 6 7 8 9 1011121314151617181920
Presentation Order
Figure 2 Mean scores - 1 standard error for the first through the twentieth list presented, arranged in order of presentation . The solid line in the figure is the curve resulting from a least-squares fit to an exponential growth function .
List Differences
The mean scores for individual lists, shown in Figure 3, ranged from 58 to 74 percent, with a mean of 64.7 percent. A one-way repeated-measures ANOVA showed a significant effect of list (F = 6.38, df = 1, 19, p < .00001). Based on key's honest significant difference post hoc analyses, the lists can be divided into two cate-gories : "easier" lists with higher mean scores, shown as solid squares in Figure 3 (lists 3, 6, 7, and 16), and the remaining 16 "harder" lists, shown as open circles in the figure. The mean scores for the easier and harder lists were 72.0 and 62.9 percent, respectively. The middle, top, and bottom horizontal lines in the figure indi-cate the mean of the 16 harder lists ± 2 SD, respectively. The post hoc analyses showed that
100 Table 1 Standard Error of Repeated Scores
90 Based on a Single List
0 80
V
d 70 - . .-0-0. . . 0 o 0
-". . . --
-~ . .
Tn o---
Source of Variability Eliminated Standard Error (%)0
60-
0
~
U 50 Subjectt None* 9.7 9.2
N 40 List and subject' 8.1
E
d
C 30 *Overall test variability with all sources of variability
O 20 included ; 'test variability with only the variability attributable to
t
a
10 subjects eliminated ; 'test variability with both list and subject0
1 2 3 4 5 6 7 8 9 1011121314151617181920
List Number
Figure 3 Mean scores ± 1 standard error for each list arranged by list number. The filled squares indicate the scores for four "easier" lists. The open circles indicate scores for the remaining 16 "harder" lists. The solid line represents the mean of the 16 "harder" lists, and the bro-ken lines represent 2 SD above and below this mean .
mean scores within the easier and harder list groups did not differ significantly from one another (p > .05). These findings suggest that selecting lists from either the group of easier or harder lists will minimize list-difference effects. Using lists from both groups, however, could lead to differences between scores that reflect interlist differences rather than differences between listening conditions .
Contribution of List Equivalency to Overall Variability
Although these list differences are statis-tically significant, it is important to consider whether these differences are substantial in light of the overall variability of the test mea-sures and the interaction between subject and list. From the ANOVA, the influence of list dif-ferences on the confidence limits of the test scores was examined by calculating the standard error of an individual subject's mean score with and without the contribution of list differences. These standard errors were calculated by first determining the sums of squares for list, sub-ject, and the list-by-subject interaction. Standard
errors were then calculated and compared for various combinations of effects . As shown in Table 1, the standard error improves by 0.5 percentage points when the variability attrib-utable to subject is accounted for. When the additional factor of list was accounted for, the variability decreases from 9.2 to 8 .1, an addi-tional 1 .1 percentage points . The slight reduc-tion in test variability with the eliminareduc-tion of
sources of variability eliminated .
the effect of interlist differences means that the test would be slightly more sensitive to dif-ferences between listening conditions. By dou-bling the standard error, one arrives at an estimate of the 95 percent confidence limits of a score. Based on the analysis described above, the 95 percent confidence limits of a score based on a single list can be reduced by 2 percentage points (roughly from -!- 18 to ± 16 percentage points) if the effect of interlist differences is removed. This effect, however, is small com-pared with the effect of using two or more lists under each condition.
DISCUSSION Practice Effects
The finding that the scores for the first list presented were lower than scores for subse-quent lists suggests that under the conditions used in the study, normal-hearing listeners need a small amount of practice before their perfor-mance stabilizes. The improvement of scores observed after the first list was presented may be a result of adaptation to the talker, adapta-tion to the noise, improved attenadapta-tion, or a com-bination of these factors. Based on the practice effects observed in this study, we recommend that one 10-word practice list be administered before collecting clinical or research data for a listener. Administration of a practice list should only require an additional minute of test time. It is important to note that the minimal practice effects observed in this study are for adminis-tration of different lists. It is possible that greater practice effects would be observed if the same list was used more than once. For example, if a lis-tener is exposed to a list under optimal listen-ing conditions and then tested uslisten-ing the same list under a more difficult condition, prior expo-sure to the list may inflate the second score.
List Differences
There are several possible explanations for the interlist differences observed in this study. One possibility relates to acoustic differences between test items in the different lists. Care was taken to control the acoustic content of the items by balancing the lists for phonemic content and equalizing the overall root mean square of the test items in recording. Nevertheless, it is possible that differences in the frequency and range of for-mant transitions made consonants easier to rec-ognize in some vowel contexts than in others. Another possibility is that linguistic factors influ-enced the difficulty of the lists. For example, items in the lists may differ in word frequency and/or neighborhood density characteristics . Words that occur more frequently in a language are known to be easier to recognize than those occurring less frequently (see Luce and Pisoni, 1998, for a review) . In addition, the number of words that are in the lexical neighborhood (i.e., that are phonetically similar to test items) has been shown to influence the likelihood that a given word will be recognized correctly (Luce and Pisoni, 1998; Meyer and Pisoni, 1999).
It is possible that a revision of the AB word lists to balance them for phonotactic character-istics, word frequency, and lexical neighborhood would eliminate the interlist differences observed here. The present findings, however, suggest that the quantitative impact on the confidence limits of a test score would be small. The reason is that these confidence limits are dominated by the interaction between list and subject. In other words, the relative difficulty of the lists differs considerably from listener to listener. By elim-inating list differences that are common to all listeners, one would improve the confidence lim-its only by a small amount.
Although the current findings indicate only a small benefit, clinicians and researchers may wish to control CASPA interlist differences by simply avoiding the four "easier" lists (3, 6, 7, 16). An alternative approach would be to apply a
correction factor to the scores obtained using the four easy lists to reduce the differences . A suit-able correction, based on the work of Boothroyd and Nittrouer (1988), would be
PC = [1- (1- Pu/100) 0.7s] " 100,
It is important to bear in mind that the findings described in this article are based on data from normal-hearing listeners tested with a specific type of noise at a single signal-to-noise ratio. It is possible that a different pat-tern of practice effects and interlist differences would be observed for a clinical population of children, older adults, and listeners with hear-ing loss. Furthermore, the effects of cochlear damage and/or hearing aid filtering are dif-ferent than those of the noise used in the cur-rent study and therefore could potentially have a different impact on both practice and interlist differences .
The findings observed in this study are but one example of how list differences and practice can affect speech recognition performance . Numerous examples exist in the literature for speech recognition tests that are commonly used by audiologists (e.g., Her Kirk et al, 1999; Bentler, 2000; Stockley and Green, 2000). These exam-ples remind audiologists that issues related to
practice effects and interlist differences need to be considered for any speech recognition
mea-sure used in clinical practice.
CONCLUSIONS
1 . Under the conditions of this study, normal-hearing adults listening in noise do not demonstrate significant practice effects after the first list is presented.
2. Under the conditions of this study, normal-hearing listeners find 4 of the lists signifi-cantly easier than the remaining 16 lists .
3. The interlist difference effect can be elimi-nated either by avoiding the easier lists or by applying a correction factor. In terms of the confidence limits of test scores, how-ever, the benefit of controlling for interlist differences will be small compared with the benefit of using more than one list under each test condition.
REFERENCES
Bentler R. (2000) . List equivalency and test-retest reli-ability of the Speech in Noise Test. Am JAudiol 9:1-17. Bentler R, Neibuhr D, Getta J, Anderson C. (1993a). Longitudinal study of hearing aid effectiveness I. Objective measures . J Speech Hear Res 36 :808-819 .
where Pc = correct phoneme recognition score in percent and Pu = uncorrected phoneme recog-nition score in percent. Note that this equation leaves scores of 0 or 100 percent unchanged.
Bentler R, Neibuhr D, Getta J, Anderson C . (1993b). Longitudinal study of hearing aid effectiveness II . Subjective measures . J Speech Hear Res 36 :820-831 . Boothroyd A. (1968a). Developments in speech audiom-etry . Br JAudiol 2:3-10.
Boothroyd A. (1968b). Statistical theory of the speech dis-crimination score. J Acoust Soc Am 43:362-367 .
BoothroydA . (1999). Computer-Assisted Speech Perception Assessment (CASPA), Version 3.0 [computer software].
Boothroyd A, Nittrouer S. (1988) . Mathematical treat-ment of context effects in phoneme and word recognition. JAcoust Soc Am 84:101-114.
Gelfand S. (1998). Optimizing the reliability of speech recognition scores. J Speech Lang Hear Res 41 :1088-1102 .
Olsen WO, Van Tasell DJ, Speaks CE. (1997) . Phoneme and word recognition for words in isolation and in sen-tences . Ear Hear 18 :175-186 .
Oruganti B . (2000) . The Effects of High-Frequency Emphasis and Amplitude Compression on the Short-Term Intensity Range of Speech . Doctoral dissertation, City University of New York, New York .
Shore 1, Bilger R, Hirsh I. (1960) . Hearing aid evalua-tion : reliability of repeated measures . J Speech Hear Disord 25 :152-170 .
Iler Kirk K, Eisenberg LS, Martinez AS, Hay-McCutcheon M. (1999) . Lexical Neighborhood Test: test-retest relia-bility and interlist equivalency. J Am Acad Audiol
10 :113-121 .
Luce P, Pisoni D. (1998) . Recognizing spoken words: the neighborhood activation model. Ear Hear 19:1-36. Meyer T, Pisoni D. (1999) . Some computational analyses of the PB-K test : effects of frequency and lexical density on spoken word recognition. Ear Hear 20:363-371 .
Studebaker G. (1985) . A "rationalized" arcsine transform. J Speech Hear Res 28:455-462.
Stockley KB, Green WB . (2000) . Interlist equivalency of the Northwestern University Auditory Test No. 6 in quiet and noise with adult hearing-impaired individuals. JAm Acad Audiol 11 :91-96.
Thornton A, Raffin M. (1978) . Speech discrimination scores modeled as a binomial variable. J Speech Hear Res 21 :507-518 .
Appendix CASPA Word Lists
List 1 List 2 List 3 List 4 List 5 List 6 List 7 List 8 List 9 List 10
ship fish thug fun fib fill badge bath hush jug
rug duck witch will thatch catch hutch hum gas latch
fan path teak vat sum thumb kill dig thin wick
cheek cheese wrap shape heel heap thighs five fake faith
haze race vice wreath wide wise wave ways chime sign
dice hive jail hide rake rave reap reach weave beep
both bone hen guess goes got foam joke jet hem
well wedge shows comb shop shown goose noose rob rod
jot log food choose vet bed not pot dope vote
move tomb bomb job june juice shed shell lose shoes
List 11 List 12 List 13 List 14 List 15 List 16 List 17 List 18 List 19 List 20
math have kiss wish hug wage jade shave vase cave
hip wig buzz Dutch dish rag cash jazz cab rash
gun buff hash jam ban beach thief theme teach tease
ride mice thieve heath rage chef set fetch death jell
siege teeth gate laze chief dime wine height nice guide
veil jays wife bike pies thick give win fig pin
chose poach pole rove wet love rub suck rush fuss
shoot rule wretch pet cove zone hole robe hope home
web den dodge fog loose hop chop dog lodge watch
