Representations of Affect

When it comes to representing emotions or affective states in synthetic sys- tems/agents, there are generally two schools of thought that have been informed by the various theories on emotions: discrete categorical labels, and continuous dimensional affect spaces. For the interested audience, there are a number of detailed and rich reviews of the issues surrounding the world of emotional rep- resentation and measurements (e.g. Plutchik (1994) and Cowie and Cornelius (2003)).

3.3.1.1 Categorical Labels

Categorical labels (e.g. “happy”, “sad”, “angry”, “scared”, etc.) are the most familiar way in which people are able to relate and refer to different affective states due to their common everyday use in natural language. Given this, these labels are self-evidently, assumed to have a coherent understudying between people and are thus the easiest ways in which to describe different emotions and states (Cowie and Cornelius, 2003), and reflects the natural tendency for people to discretise their sensory input froth surrounding world into manageable chunks as outlined by James (1890). In the majority, focus of affective labels has been around what has been termed the “basic six” emotions (Schr¨oder, 2001; Plutchik, 1994; Scherer,

1986; Banse and Scherer, 1996; Cowie and Cornelius, 2003), which has primarily been due to the prominent theories surrounding the notion of basic emotions: happiness, sadness, surprise, fear, anger and disgust (e.g. Ekman (1992) and Izard (2007)).

With respect to measuring emotion from humans and representing emotions in affective systems, there are a number of drawbacks. Firstly, given the links with natural language, the use of linguistic labels of measurement requires caution as, specifically in cases dealing with emotional human speech, these can serve as a considerable bias as the stimulus can carry semantic linguistic information regarding the emotional labels (Plutchik, 1994).

There is also an issue of resolution: emotional labels inherently do not provide a granular measure or indication how intense an emotion is. They are not able to capture subtle, but important differences between affective states. In natural language, a listener is able to utilise a variety of different cues regarding this through the nature of multi-modal interaction.

Finally, there is the issue of the number of labels that is to be used during measurement. In the case where the are only a few labels, which has been a common practice in a number of fields, the rating of stimuli is more akin to a discrimination task rather than an identification task (i.e. subjects are more likely to provide ratings based upon what the stimulus is not, rather than focusing upon what it is), as Banse and Scherer (1996) and Scherer (1986) have highlighted. This can be overcome by introducing many more affective labels (Schr¨oder, 2001), however this can make the experimental process notably longer, but has had the benefit of allowing assessment of the how many different affective labels can be broken down into more fundamental underlying components such as affective dimensions, as demonstrated by Russell (1980) with the Circumplex model of emotions.

With respect to their representation in synthetic systems, emotional labels have the benefit in that each affective state that is modelled can have an acti- vation level, which allows multiple affective categories to be active at the same

time, something that has been shown to be useful in the design of systems that recognises and represent multiple complex mental states from the human face for example (Kaliouby and Robinson, 2004). However, the inherent lack of granu- larity is also a problem in that it means that in the eyes of recognition systems, people can “jump” from state to state, which is not representative of the how the behaviour or mental state of a person changes as an interaction unfolds. Further- more, in systems that are designed to express affect or just act upon it, categorical representations of affect with respect to the modelling of an input, or the inter- nal state of the system itself can lead to large changes in behaviour due to the tendency to also jump between affective states, which is also generally undesired (Schr¨oder, 2003b).

3.3.1.2 Dimensional Affect Spaces

Dimensional representations seek to identify ways in which emotional/affective states may be represented in continuous manner in spaces that have a small num- ber of dimensions. There are multiple facets that make this approach appealing not only to the field of psychology, but also to fields concerned with creating syn- thetic systems that deal with affect (for example, the field of Affective Computing (Picard, 1997), and HRI (Breazeal, 2002)). For example, one of the main attrac- tions is that dimensions provide a way in which affective states can be described in a more tractable manner, but can also be translated into and out of common verbal descriptions commonly used by people (Fontaine et al., 2007). This trans- lation is possible as emotion related words can be mapped to different affective dimensions (e.g. Russell (1980)), and thus referred to specific locations within these dimensions (Cowie and Cornelius, 2003). Thus, dimensions are able to not only capture subtle differences in affect to a high resolution, but it is also possible to interpret the dimensions into more coarse regions which can form the basis of a categorical representation also (Schr¨oder, 2004), making them useful when investigating what effects subtle changes to a stimulus (e.g. an emotional face, or a vocal utterance) has upon how people affectively interpret these (Cowie and

Cornelius, 2003). Furthermore, given that dimensions provide a numeric repre- sentation, they lend themselves to world of machine learning, which exploits a variety of mathematical tools used to manipulate numeric data.

This approach however is not without problems and shortcomings. Firstly, and perhaps more importantly, is that as with the basic emotion theories, there are disagreements with respect to both the number of dimensions an affect space should consist of, but also what the different dimensions represent. This is a prac- tical problem in that in situations where there are only two dimensions, certain states such as Fear and Anger, and Excitement and Surprise are difficult to dif- ferentiate (Fontaine et al., 2007; Zeng et al., 2009). As such, this has resulted in a large number of different affect spaces, with ongoing debate as to which spaces are most optimal. An issue that still remains very much open (Cowie and Cornelius, 2003).

A further drawback is that in dimensional spaces, only a single affective state can be modelled at a given moment in time, whereas with categories, the number of states is determined by the number of categories, each of which can have a self contained level of activation. This means that if any co-occurring affective states arise simultaneously, only one of these may be represented in the affect space.