Study 2: Music

Similarly to Study 1, there were no significant effects of stimulus melody or instru-mentation, therefore these variables were collapsed for subsequent analyses. Thus, participants’ scores were summed across these variables to produce only one value for each unique combination of: Acoustic feature processing condition× Emotion × CI condition. As in Study 1, judgement scores (correctness multiplied by confidence rating) varied between -20 to 20 for the following analyses. Prior to running further analyses, it was confirmed that these judgement scores were normally distributed.

As in Study 1, chance level performance was estimated as corresponding to -7.2.

For each of the four melodies in each condition (after collapsing across melody and musical instrument), the odds of choosing the correct emotion by chance were 20%

(since there were five response options), while the odds of choosing an incorrect emotion were 80%. Multiplying these odds by response ‘correctness’ and the assumed confidence rating of 3, chance performance for the judgement score was 3(.2× 1 × 4)− 3(.8 × −1 × 4) = −7.2.

Overall, participants’ average reaction times in Study 2 (global mean RT from stimu-lus onset = 8.47, or mean stimustimu-lus length + 1.84 seconds) were very similar to Study 1 (mean RT = 5.04, or mean stimulus length + 1.92 seconds), once differences in stimuli lengths were taken into account. However, confidence ratings were on aver-age half a point lower in Study 2 (mean = 2.89, compared to 3.44), suggesting that participants had greater difficulty in decoding emotion from musical stimuli.

A three-way repeated measures ANOVA revealed significant effects of both auditory feature processing condition, F (4, 56) = 4.19, p = .005, η²_p= .23, and emotion, F (4, 56) = 6.35, p <.001, η²_p = .31 upon participants’ judgement scores. However, the effect of CI condition upon judgement scores was not significant, F (1, 14) = 3.66, p

= .076, η_p²= .21, nor was the interaction between CI condition and Auditory feature processing condition, F (4, 56) = 2.44, p = .058, η_p²= .15.

Nonetheless, relative to the Original condition, judgement scores in the Duration conditions were affected to a noticeably lesser extent by the CI manipulation, whilst scores in the Articulation, Frequency and Intensity conditions all showed at least some deterioration as a consequence of CI simulation (Figure 19). Notably, compared to the results of Study 1, scores were generally negative, reinforcing the observation that participants found this task much more difficult with music stimuli. It should be noted, however, that scores were mostly above the estimated level of performance expected by chance (-7.2).

Articulation Duration Frequency Intensity Original

Judgment score (-20 to 20)

Processing condition

Without CI With CI

Figure 19: Interaction between auditory feature processing condition and CI con-dition, for participants’ judgement scores. Error bars denote standard error of the mean.

The ANOVA also revealed a significant interaction between Emotion and CI

condi-tion, F (4, 56) = 7.34, p <.001, η²_p = .34 (Figure 20). Planned contrasts determined that, compared to Neutral stimuli, judgement scores for Anger and Fear were af-fected significantly differently by the CI manipulation (i.e. performance was better, rather than worse, with CI simulation), whereas scores in the remaining emotional conditions were not (Table 6).

Anger Fear Happiness Neutral Sadness

Judgment score (-20 to 20)

Processing condition

Without CI With CI

Figure 20: Interaction between Emotion and CI condition, for participants’ emotion judgement scores. Error bars denote standard error of the mean.

Table 6: Planned contrasts illustrating how participants’ judgement scores for the different emotions, relative to the Neutral stimuli, were affected by the CI manipu-lation.

Lastly, the ANOVA revealed a significant three-way interaction between emotion, auditory feature processing condition and CI condition, F (16, 224) = 2.76, p <.001, η_p² = .17 (Figure 21). Planned contrasts were computed to inspect this relationship in more detail (Table 7). In the Articulation condition, relative to the Original

condi-tion, judgement scores were affected significantly differently by the CI manipulation only for Happiness and Sadness stimuli, compared to Neutral. In the Duration and Frequency conditions, judgement scores were affected significantly differently by the CI manipulation only for Happiness stimuli. By contrast, judgement scores were af-fected significantly differently by the CI manipulation for Anger, Fear and Happiness stimuli in the Intensity condition. In summary, without CI simulation, in the Origi-nal and Frequency conditions Happiness and Sadness were by far the emotions most often identified, and therefore identification accuracy in these particular conditions appeared to be especially affected by the CI manipulation.

To examine more closely the patterns of errors made by participants during the emotion judgement task, confusion matrices are shown below for each of the audi-tory feature precessing conditions, and for both non-CI-simulated and CI-simulated conditions (Figure 22).

Foremost, not all of the emotions were consistently decoded, with Anger and Fear stimuli never reaching above 40% recognition accuracy (irrespective of the feature-attenuation and CI simulation conditions), in contrast to the results reported by Quinto et al. (2014) with the same stimuli. In fact, Anger and Fear were seldom chosen as responses, while Neutral was overestimated – most likely an index of general uncertainty.

As with speech stimuli in Study 1, without CI simulation, the Frequency condi-tion produced the pattern of errors most similar to the Original stimuli. In these conditions, relative to Articulation, Duration and Intensity, participants tended not to overestimate the Neutral stimuli, suggesting that participants perceived these stimuli as being more expressive. However, specific emotions were again affected dif-ferentially by the feature attenuation conditions. For example, Sadness was equally

-15

Articulation Duration Frequency Intensity Original

Judgment score (-20 to 20)

Processing condition

Anger Fear Happiness Neutral Sadness

-12

Articulation Duration Frequency Intensity Original

Judgment score (-20 to 20)

Processing condition

Anger Fear Happiness Neutral Sadness

Figure 21: Interaction between auditory feature processing condition, emotion and CI condition, for participants’ emotion judgement scores. Upper panel = non-CI stimuli, lower panel = CI stimuli. Error bars denote standard error of the mean.

well-decoded in both the Intensity and Frequency conditions, whilst Happiness was decoded much better in the latter condition. As in Study 1, Sadness stimuli were on the whole the most robust to distortion, but appeared to be better-preserved in the Duration condition.

The matrices associated with the CI-simulated stimuli showed a somewhat

differ-Figure 22: Heat-mapped confusion matrices for the music stimuli, depicting the per-centage of responses in each emotion category, for each stimulus emotion. Columns denote presented emotions, rows denote emotion judgement responses. Red cells = higher values, green cells = lower values.

ent, more complicated pattern. Overall, emotions were decoded most accurately in the Original condition. Elsewhere, judgement accuracy varied heavily depending upon both the emotion and the feature-attenuation condition. Happiness was well-decoded in the Frequency condition, but otherwise was often confused, particularly with Anger. Conversely, Anger was decoded with greater accuracy in the Intensity and Duration conditions, even compared to the Original condition. Fear was also relatively well-recognised in the Intensity condition, although only 30% accuracy was achieved.

There were also some more general response biases caused by the CI simulation.

By comparison to the non-CI-simulated stimuli, Anger and Fear were much more readily offered as responses in the CI simulation condition. By contrast, Sadness was a far less common response in this condition. As with the non-CI simulated stimuli, Neutral stimuli tended to be overestimated, especially in the Articulation, Duration and Intensity conditions.

As in Study 1, there were no significant relationships between participants’ emotion judgement scores and any measures of musical training or musical engagement. There was also no significant relationship between judgement scores and participants’ self-reported empathy scores.

Table 7: Planned contrasts illustrating how participants’ emotion judgement scores were affected by the CI manipulation in the different auditory feature processing conditions, relative to the Original condition, for each of the different emotions.

Condition Emotion F df p ηηη²_p²_pp²

Articulation

Anger 2.67 1, 14 .125 .16

Fear 3.36 1, 14 .088 .19

Happiness 30.87 1, 14 <.001^* .69

Sadness 5.58 1, 14 .033^* .29

Duration

Anger 1.66 1, 14 .22 .11

Fear 1.18 1, 14 .296 .08

Happiness 11.51 1, 14 .004^* .45

Sadness 2.63 1, 14 .127 .16

Happiness 25.25 1, 14 <.001^* .64

Sadness 3.73 1, 14 .074 .21

*= significant at .050 alpha-level.

4.7 Discussion