How is emotion communicated and perceived in speech and music?

To understand how listeners go about perceiving and identifying an emotion conveyed by an audio stimulus, it is useful to begin by considering how individuals approach perceptual identification tasks more generally. Within cognitive psychology,

cogni-tive neuroscience, and their related fields, the most prevalent account explaining the processes of discrimination and identification (e.g. of different emotions) is the in-formation processing approach (Lindsay & Norman, 1977). Broadly speaking, this perspective assumes a several-stage process of analysis, matching and association – a sound is transduced, informative properties of the auditory signal are extracted, and these are then compared with stored representations in long-term memory, in order to assign semantic significance (McAdams, 1993).

Although often referred to as one complete process, it is worth noting that listening to a piece of music and ascribing an emotion to it may be described in terms of two different processes: perception and recognition. Where such a distinction is made, perception refers to a series of lower-level processes, concerned with the sensory trans-duction of a stimulus, whereas recognition denotes a process by which meaningful attributes are assigned to this stimulus, typically by comparing what is perceived to some stored mental representation (Adolphs, 2002). Therefore, recognition may be said to depend on information external to the stimulus, at least to a greater extent (discounting, for the sake of simplicity, constructivist accounts of perception, which would claim that the incorporation of top-down information is an essential part of perception itself (Gregory, 1970)). Strictly speaking, some experimental psycholo-gists have drawn additional distinction between recognition and identification, where the latter task constitutes a more narrowly-focussed variant of the former (McAdams, 1993). Specifically, identification requires that stimuli be labelled according to some lexicon of different names – in this case, emotions – whereas recognition may denote mere familiarity with a stimulus (i.e. seen vs. unseen). Therefore, the most com-monly used paradigms in the study of emotion perception can usually be referred to as identification tasks.

Considering the case of emotion identification, and assuming that we want to cat-egorise a perceived emotional stimulus as one or more of several emotion cate-gories, there are two predominant strategies available: rule-based categorisation and similarity-based categorisation (Smith, Patalano, & Jonides, 1998). In the former, rules of the form ‘if X then Y ’ are applied to a set of stimuli, where X denotes some particular feature configuration, and Y denotes the resultant classification. As an example, one might learn that, if a piece of music is composed in a major key, then it conveys happiness. In fact, in the field of artificial intelligence, this rule-based approach was at the heart of the ‘expert systems’ popular in the 1970s and 1980s, in which domain-specific expert knowledge is implemented computationally by a series of if-then rules (Russell & Norvig, 1995). By contrast, similarity-based categori-sation entails a computation of the similarity of each stimulus-to-be-classified with stored exemplars of the various emotional categories. In particular, relevant stimulus parameters are extracted and compared with acquired knowledge about the distri-butions of these parameters in each of the categories-to-be-classified. This method is analogous to more modern machine learning approaches to classification, for example involving the use of neural networks (Galushkin, 2007).

For the task of emotion recognition, similarity-based classification is likely to pre-dominate for two key reasons. Firstly, owing to the complex and multidimensional nature of emotional stimuli, it is unlikely that simple rules would have much more predictive power (Smith et al., 1998). That is, a set of rules such as ‘if the stimulus is very loud, then the emotion communicated must be anger’ would be too inflexible and would likely encounter too many counterexamples to be effective. Secondly, with increased experience, individuals tend to gravitate towards similarity-based categori-sation. This approach inherently becomes more accurate when more exemplars are

available for comparison (Allen & Brooks, 1991), and is also typically faster and less cognitively demanding (Smith and Kemler, 1984; Smith and Shapiro, 1989). Since most individuals will have had extensive experience in perceiving and recognising myriad emotional states, it seems fair to assume that they will have long abandoned rule-based categorisation.

It should be noted that the Information Processing account of emotion perception and recognition is not necessarily as fixed or linear as the above (simplified) account implies. In reality, situational factors, individual differences, task effects and per-sonal goals may all interact and exert influence upon perception and identification.

Additionally, higher-order ‘meaning’ may not necessarily arise simply from compar-ison of a stimulus to some stored representation(s), but instead may be constructed or inferred based on various different items of existing experience and/ or knowledge.

To summarise, in order to perform well in auditory emotion identification tasks featuring speech or music, most individuals will: gather perceptual information by listening, organise this information in terms of the relevant acoustic features, and then compare observed patterns of features with stored representations acquired via previous experience. Of course, this process requires some awareness (though perhaps not necessarily a conscious awareness) of the ways in which emotions are usually conveyed in speech and music. In other words, an individual will need to have already experienced a wealth of emotional speech and music, in order to determine which acoustic parameters are most relevant, and which configurations of these parameters tend to represent particular emotions. Accordingly, the next section is concerned with exactly this: establishing which auditory features are used for the communication of emotion in speech and music, and how these features interact in order to convey distinct emotional states.