Capturing Affect from People

Capturing affect from human subjects comes in two flavours: implicit and explicit methods (Broekens et al., 2010; Isomursu et al., 2007). Implicit methods measure behavioural characteristics of a person (heart rate, respiration rate, skin resistance, etc, see Picard (1997) and Zeng et al. (2009) for overviews), while explicit methods require that subjects self report and input data directly (suggesting or choosing emotional labels or adjectives, selecting an emotional face, etc.). The body of work presented in this thesis has solely employed the latter, as not only would the use of implicit measures introduce a cumbersome aspect to the experimental process due to the requirement for placing many sensors in various locations on the body5_,

5_{This has some very sensitive ethical issues associated with it when experimentation with} children is conducted.

particularly with young children, but also it limits the amount of comparison that may be made the the related literature on both NLUs and gibberish speech in social agents (none of the previous work has used physiological measures of affect), but also work in affective expression via the human voice and music.

The previous section outlined the two main approaches that have been estab- lished with respect to how emotions can be represented in synthetic systems that have an affective component: categorical labels and affective dimensions, and dis- cussed their respective benefits and drawbacks. These representations affect have also fed into the design of methods for how affect can be measured and captured from people.

A final noteworthy point to highlight is that while affective dimensions have a number of properties that make them appealing, particularly with respect to the goal of addressing the research questions outlined in chapters 1 and 2, measuring tools based around emotional categories are easy to explain to naive subjects both young and old, and tools based around affective dimensions are not. This is a weightily point to consider as some of the experiments undertaken in this thesis have been performed with children. As a result, time and care has been taken to familiarise the subjects with the measuring tools used, and to confirm that they indeed were able to the measuring tools in an appropriate manner.

There are a number of different tools that have been developed for affec- tive measurement based around affect dimensions, namely the Self Assessment Manikin (SAM), (Lang and Bradley, 1994), FEELTRACE (Cowie et al., 2000), EMuJoy (Nagel et al., 2007) and the AffectButton (Broekens and Brinkman, 2009; Broekens et al., 2010; Broekens and Brinkman, 2013).

The SAM is a tool is a picture orientated tool that is designed to assess the Pleasure, Arousal and Dominance dimensions independently. Graphical images are shown to depict major points along each dimension. For the pleasure dimen- sion, the images shown an agent with differing facial gestures ranging from a large happy smile to an unhappy frown. Arousal is depicted with a figure with a wide- eyed excited face to a sleepily and relaxed face. Dominance is shown with the

figure with varying physical size, which relate to the amount of control that the figure has with respect to the surrounding environment (the surrounding box in this case): a large figure translates to high control and thus dominance, while a small figure translates to the figure having little control.

FEELTRACE is a tool that was specifically designed for capturing peoples’ affective ratings of emotional speech. It shows a two dimensional Activation- Valence space, with a number of different affective labels located in the space, to help guide the users understanding of the dimensions and how they relate to the affective labels that they are familiar with through the use of their natural language vocabulary. A coloured circular icon is controlled by a computer mouse and dynamically changes colour as it is targeted around the space following a specification of colour/affect mappings as proposed by Plutchik (1994). This tool also has a functionality that allows previous inputs to be stored and shown graphically so as to capture a history of affective measurements for a subject. This is partially useful when presenting subjects with stimuli that span across different time scales and that require more than just a snapshot in time measure.

EMuJoy is a tool that has been developed for capping peoples’ affective ratings of musical pieces rather than speech. Though compared to FEELTRACE, that was designed for the measurement of affective speech, these two tools actually have a very similar end solution. With EMuJoy, again two dimensions are shown on screen, Arousal and Valence, with a cursor that shows the current position the two-dimensional affect space. The cursor takes the form of a small expressive face that dynamically changes as the cursor is moved around the input space, to represent the general affect of the current location. This tool also facilitates a history of affective measurements in the form of a worm tail which shows the previous inputs by the user in their chronological order.

The AffectButton is a tool that shows only an expressive face that changes dy- namically as the mouse cursor is moved around the input space. Each face is also encoded into a three-dimensional coordinate where the dimensions correspond to Pleasure, Arousal and Dominance. What is unique about this tool compared to

Figure 3.7: The AffectButton prototype facial expressions with PAD values. From left, clockwise: Neutral (0,-1,0), Angry (-1,1,1), Excited (1,1,1), Scared

(-1,1,-1), Surprised (1,1,-1), Annoyed (-0.5,-1,0.5), Happy (0.5,-1,0.5), Sad (-0.5,-1,-0.5), Content (0.5,-1,-0.5). Adapted from Broekens et al. (2010).

the others outlined above is that the underlying affective dimensions are com- pletely hidden from the subject, and thus there is no need to even mention the notion of affective dimensions to users. This is a key benefit (as discussed in the next section), and is why this tool was selected as the affective measuring tool of choice during the experiments presented in this thesis.