2.4 Communicating Affect through NLUs and gibberish speech: re-
2.4.3 Discussion
As can be seen from the review of work directly related to HRI, research focusing NLUs and gibberish speech as been very limited indeed. While on one hand this is a concern as there is little directly relevant work to relate to, the positive aspect is that it leaves many open questions that need to be addressed. Some of these are discussed here, as they have influenced the direction that the work in this thesis has taken and the methodology that has been adopted. These are the overall methodology used for evaluation affective meaning in NLUs, the evaluation settings and subjects used, and the need to go beyond only evaluating different methods for creating utterances, but rather explore how real world HRI influences how NLUs should be used and what impacts this.
2.4.3.1 Evaluation Methodology
Generally, throughout both the work on NLUs and gibberish speech there has been a tendency toward studies using a decoding paradigm (Banse and Scherer, 1996; Scherer, 2003), where a limited number utterances have been created to represent a limited number of affective categories/labels (e.g. anger, happiness, ect), and subjects are to decode each utterance and assign one of the labels to the utterance. Table 2.3 seeks to illustrate this and charts the various studio that have been referred to in this review, outlying the method of utterance generation, the acoustic parameters that were varied, and the difference affective states that were portrayed by the utterances during evaluations and the means of measurement of subject interpretations. As we can see, most of the studies have employed discrete affective categories as the measurement method.
These studies often report confusion matrices in their results (e.g. (Breazeal, 2002; Oudeyer, 2003; Tuuri et al., 2011)), indicating which states are confused
with others. While through this it is possible to identify confusion patterns be- tween states, leading to potential insights regarding how close different concepts of affective states are, such information provides limited insights into what features of each sound were involved in the users decoding process, and to what degree these utterance features influence affective charging.
This method of evaluation also brings up a larger debate surrounding the use of categorical measurement as a whole (particularly in the speech synthesis do- main): how many different categories should be used? Banse and Scherer (1996) and Scherer (2003) argue that with fewer forced choice categories presented to subjects, evaluation becomes more a task of recognition rather than discrimina- tion. They take the view that simple, basic emotion categories (Ekman, 1992; Ortony and Turner, 1990; Plutchik, 1994) are better broken down into pairs, such a hot and cold anger, providing higher granularity in affective measurement and increases the number of labels presented. This is an ongoing debate that has not be resolved as of yet, and is likely to be very dependent upon the subjects that are being tested (e.g. adults v.s. children). While such decoding studies are interesting insights from the perspective of probing emotional representations, a general drawback is that this approach is less fruitful when one is concerned with understanding the mechanisms through which utterance features are exploited to charge an utterance. These are vital insights when one in concerned with under- standing how utterance features are best exploited to express particular states, and if one is to introduce variability to this.
There is also a debate surrounding whether categories are the most appro- priate representation of affect to present to subjects. There are suggestions that measurement tools based upon dimensional representations of affect are a more suitable foundation to employ since continuous dimensions are more resolute to subtle changes in affective states (Cowie and Cornelius, 2003; Schr¨oder, 2004; Laukka, 2005), changes that are likely to occur during social interactions. Further more, if one were to use dimensional approaches for measurement, one becomes well poised to gain insights about how subjects actually perceive the stimuli. Do
they exhibit coarse differences in affective inference, or are the inferences subtler? Unfortunately, there is also the question as to the number of dimensions there should be in such an affect space and what each dimension represents (Fontaine et al., 2007). This is something that shall be addressed in more detail in the next chapter.
There appear to have been no inference studies thus far. Rather than be- ing concerned with a users ability to recognise and decode the affective state of an agent based upon utterances designed to convey certain emotions, inference studies are more concerned with uncovering the underlying mechanisms for each acoustic feature of an utterance that influence affective expression, communication and interpretation (Scherer, 2003). This somewhat reflects the strategies through which authors of the previous work have affectively charged their utterances. In the majority of cases, studies have drawn upon insights from previous work across various fields to inform the design of stimuli. While this approach is solid in that it draws from previous results, it also makes the assumption that there is enough overlap between parameter modelling and configuration of previous work and the current work. This is more likely the case with gibberish speech that this assump- tion stands than with NLUs, given the heavy reliance of TTS technology and the use of human voice recordings. Not all studies have adopted this approach however. Affective charging has also been achieved via simulating affective states by recording actor portrayals and using these as templates. However there are questions over how genuine actor portrayals of emotions actually are (Scherer, 1986; Banse and Scherer, 1996; Scherer, 2003).
2.4.3.2 Subjects and Evaluation Settings
Another notable observation form the previous work is that the general age range of subjects has also been somewhat constant across all the studies that have been referred to (see table 2.4). All evaluations have been with adult subjects, and been conducted within a lab setting, with the exception of the work by Yilmazyildiz et al. (2013) who performed their evaluation with both adults and young teenagers.
Table 2.3: Acoustic parameters used to generate utterances in previous work, the emotions that were portrayed and the means of affective measurement.
Study NLU/Gibberish Acoustic Parameters Emotions Portrayed Measurement
Breazeal (2002) Gibberish
Accent Shape Categories: Categories:
Average Pitch Anger Anger
Contour Slope Fear Fear
Final Lowering Disgust Disgust
Pitch Range Happiness Happiness
Pitch Base Surprise Surprise
Speech Rate Sorrow Sorrow
Stress Frequency Neutral Neutral
Breathiness Brilliance Larypngealization Loudness Pause Discontinuity Pitch Discontinuity Precision of Articulation Oudeyer (2003) Gibberish
F0 mean Categories: Categories:
F0 variance Happiness Happiness
F0 max Sadness Sadness
F0 contour Anger Anger
Last word contour Comfort Comfort
Last word accent Calm Calm
Accent probability Mean duration Duration variance Volume
Jee et al. (2007) NLU
Tempo Categories: Categories:
Key Happy Happy
Pitch Sad Sad
Melody Fear Fear
Harmony Dislike Dislike
Rhythm Volume
Jee et al. (2009) NLU
Tempo Categories: Categories:
Key Joy Joy
Pitch Distress Distress
Melody Shyness Shyness
Harmony Irritation Irritation
Rhythm Pride Pride
Volume Dislike Dislike
Expectation Expectation
Anger Anger
Jee et al. (2010) NLU
Intonation Categories: Categories:
Pitch Range Happiness Happiness
Timbre Sadness Sadness
Affirmation Affirmation Denial Denial Encouragement Encouragement Introduction Introduction Question Question Komatsu (2005) NLU
Pitch Frequency Categories: Categories
Frequency Envelope/Pitch Slope Disagreement Disagreement
Duration Hesitation Hesitation
Agreement Agreement
Komatsu and Yamada (2007, 2008); Komatsu et al. (2011) NLU
Pitch Frequency Categories: Categories
Frequency Envelope/Pitch Slope Positive Positive
Duration Negative Negative
Undistinguishable Undistinguishable
Komatsu et al. (2010); Komatsu and Kobayashi (2012) NLU
Pitch Frequency Robot’s Confidence: Perception of Confi- dence:
Frequency Envelope/Pitch Slope Confident Confident
Duration Not Confident Not Confident
Tuuri et al. (2011) NLU
Pitch Contour Categories: Categories:
Utterance Duration Slow Down Slow Down
Voice Intensity Urge Urge
Ok Ok
Reward Reward
Yilmazyildiz et al. (2006) Gibberish
Pitch Categories:
No evaluation took place.
Timing Anger, Joy
Sadness, Fear
Yilmazyildiz et al. (2010) Gibberish Mary TTS parameters Categories: Categories:
Happy, Sad Happy, Sad
Yilmazyildiz et al. (2011) Gibberish
Recorded actor samples were used Categories: Categories:
Neutral Neutral Anger Anger Disgust Disgust Fear Fear Happiness Happiness Sadness Sadness Surprise Surprise
Yilmazyildiz et al. (2013) Gibberish
Recorded actor samples were used Categories: Categories:
Neutral Neutral Anger Anger Disgust Disgust Fear Fear Joy Joy Sadness Sadness Surprise Surprise
Also, the number of subjects has generally quite low. It appears that there has been no work probing the interpretation of NLUs by young children (ages less than 10). Another interesting observation is that there have been no studies that have addressed subjects with social disorders such as autism, while efforts have been undertaking into understanding how robots can be used to investigate such social disorders (e.g. Robins et al. (2004)).
Child-Robot Interaction (cHRI) is an area of HRI that has shown great promise in recent years and is currently gathering momentum as a subfield of HRI as evidence through research efforts such as the ALIZ-E project (Belpaeme et al., 2012). The primary reasons for this are the willingness that children shown to engage in HRI, and suspect their disbelief (Breazeal, 2003a; Ros Espinoza et al., 2011; Salter et al., 2008).
Given the increasing number of potential application areas of cHRI, NLUs may have a particular amount of promise for the the use in this area, however there have currently been no efforts to explore this as of yet. This is one particular facet that the work in this thesis seeks to address, if only initially. Furthermore, there has been no work addressing how adults and children differ in their perception of NLUs and whether they have the same affective inferences from utterances. This too is something that this research seeks to address where possible.
2.4.3.3 Going beyond Affective Charging
The vast majority of the previous work has focused upon how NLUs and gibberish speech can be generated, presenting a variety of different methods and techniques. A limited number of utterances are then created using these methods, designed to convey a given affective state. Unfortunately, these studies become evaluations of specific methods for creating utterances which means that the results have a limited capacity for generalisation as to the application of NLUs and gibberish speech more broadly. Furthermore, some of these evaluations were conducted without the use of a real robot (e.g. Oudeyer (2003), Jee et al. (2007), Yilmazyildiz et al. (2006), Yilmazyildiz et al. (2010) and Yilmazyildiz et al. (2011)). The
Table 2.4: Number of subjects and subject age ranges in reviewed previous work.
Study # Subjects Age Range
Breazeal (2002) 9 23-54
Jee et al. (2007) 20 “undergraduates”
Jee et al. (2009) NA NA
Jee et al. (2010) 20 20-25
Komatsu (2005) 23 20-42
Komatsu and Yamada (2007) 9 21-24
Komatsu and Yamada (2008) 20 19-24
Komatsu et al. (2010) 19 22-25
Komatsu et al. (2011) 20 19-24
Komatsu and Kobayashi (2012) 20 21-28
Oudeyer (2003) 8 “adults” Tuuri et al. (2011) 12 23-29 Yilmazyildiz et al. (2006) NA NA Yilmazyildiz et al. (2010) 10 24-37 Yilmazyildiz et al. (2011) 11 27 - 32 Yilmazyildiz et al. (2013) 35 10 - 14
problem is that such evaluations focused upon a specific technique for creating utterances has no real input regarding how NLUs/gibberish speech can be used during real HRI.
This is something that sets the works by Komatsu et al. aside. Their re- search has focused upon a very simple set of utterances, but explored how these utterances are perceived by people when presented through different robots, and more importantly, how these utterances influence a real interaction that is not context free. Such knowledge something that the overall field is lacking and is in desperate need of if both NLUs and gibberish speech are unlock the potential benefits that they have to offer during real world HRI. The results of obtained from their simple experiments with simple utterances have been fascinating and provided initial valuable insights showing that within a interaction, NLUs indeed can have an important influence over how people behave.
However, the drawback with their work is that they have employed very sim- plistic utterances that have been hand crafted and have no really addressed af- fective meaning beyond simple positive or negative valence. The world of vocal affective displays is far richer than this, and a clear area that requires further exploration is how more acoustically rich NLUs are capable of conveying more
complex affective states, how these can be used during HRI, and how these can influence people during an interaction as well as how the interaction can influence the utterances. Highlighting and addressing such potent questions is another the prime goal of the research presented in this Thesis.