In the future, it is interesting to see if presenting more behaviors of a certain attitude in one fragment would actually result in more agreement on the judgments of the attitudes. If it does, different levels can be set to the interrogation game by adding different postures. For instance, if the intended stance is dominant and more dominant postures or expressions can be added, then it is an easy level. By setting levels to the game, it will be possible to verify if the interrogation game actually helps police trainees with their ability to grasp the social signal conveyed by suspects’ behaviors. However adding planned posture or expression in real-time introduces complexity in synchronization of the speech, posture and facial expression, which is essential for the genuine believability of virtual humans. A sophisticated computational model of behavior planner is required.
In order to build a computational model of behavior planner, the interaction pattern between the police and the virtual suspect should be further investigated. According to [16] and [8], the interpersonal attitude is a combination of the personality of the person and their relationship to the other person. The response of an interpersonal attitude can take two forms, compensation and reciprocation [16]. Compensation means that the other person responds in an opposite way or attitude, for example, if one person takes a close posture the other reacts with a distant posture. The same compensation also holds for dominant behavior, which may result in a submissive attitude of the other person. However, people may also respond in a reciprocal manner, for example, during conflicts, a dominant—hostile behavior may lead to dominant—hostile behavior of the spouse [16].
The behavior planner should present a combination outcome of personality and attitude [16, 3], since personality traits have a major influence on social attitude [8]. Based on the posture generation process in [16], the behavior generation model in Figure 16 is proposed.
In an interactive system, users can choose their response to the conversational agent. The combination of user input, agent’s personality and current attitudes (affiliation and controlling) determines the new attitude. The scheme of interaction pattern between the police and the virtual suspects should be applied when generating new attitude. The new attitude immediately becomes the current attitude. Interpersonal attitude can be conveyed through different modalities. Depending on the agent’s personality, agents have different behavior types to show their attitudes. There are six behavior types, which are close (high affiliation), distant (low affiliation), space filling (high controlling), shrinking (low controlling), relaxation (high controlling) and nervousness (low controlling) [16]. The behavior type gives feedback to user input, so
38
users have options to choose from. The concrete behavior is also determined by behavior types.
Figure 16. the behavior generation model based on [16]
In order to have a more systematic and comprehensive facial animation system, the system can be further adjusted according to Facial Action Coding System (FACS) [13] by matching the blending shapes to Action Units. There are 46 action units in FACS. Each action unit can be simulated using blend shapes, so all the facial expression in FACS can be simulated by different combination of blend shapes.
Agent’s Personality User Input New Affiliation Current Controlling Current Affiliation New Controlling Behavior Types Concrete Behaviors
39
Summary
The purpose of this research is to investigate how people perceive the stances expressed by the virtual humans. In this report, we presented an interrogation environment that is capable of generating believable behaviors in real-time. Pose libraries were built for each attitude. One can import their own 3D avatars into the system, if the avatars have the same layout of the blend shapes. These avatars can be built in 3D modeling software such as Blender, Maya, and Cinema 4D.
The result of the user tests reveals that people tend to categorize the virtual suspects into the same attitudes as human actors and audio helps people judge suspects’ attitudes. There are weaknesses in this study. The objects may have an influence on people’s perception of suspects’ attitudes. Even though the result of user test II indicates no significant difference in people’s judgments after removing the objects, the number of participants is too small to ensure the conclusion. Another weakness is the lady suspect who introduces more than one stereotype. For parallel comparison, the lady suspect should be replaced at least with a male avatar.
There is still work to be done in order to fulfill the goal of the project that aims at building conversational agents for interrogation games in which police trainees can train interrogation strategies. In the future, a computational model that encompasses behavior generation scheme as in Figure 16 should be developed. The interaction pattern between the police and the suspects is still to be determined. Formulas which give weights to the user input, agent’s personality and current attitude when calculating the new attitude should be introduced. Integrating the behavior generation model with the interrogation environment, we can have a conversational agent that function as a virtual suspect in a game.
40
Bibliography
[1].Argyle, Bodily communication, 2nd edition. Methuen, New York , 1988. [2] N. Ambady and R. Rosenthal, Thin Slices of Expressive Behaviors Predictors of Interpersonal Consequences. A Meta-Analysis: Psychological Bull, vol. 111, no. 2, pp. 256-274, 1992.
[3] I. Ajzen, Attitudes, Personality and Behabior. McGraw-Hill International, pp 34, 2005
[4] T. W. Bickmore, Relational agents: Effecting change through human-computer relationships. Doctoral dissertation, Massachusetts Institute of Technology, 2003. [5] T. W. Bickmore and R. W. Picard, Establishing and Maintaining Long-term Human Computer Relationships. ACM Trans. Comput.-Hum. Interact., 12(2): pp.293–327, 2005.
[6] G. Beattie and H. Shovelton, An Exploration of the Other Side of Semantic
Communication: How the Spontaneous Movements of the Human Hand Add Crucial Meaning to Narrative. Semiotica., 184, pp.33-51, 2011.
[7] D. Carney, J. Hall, and L. LeBeau, Beliefs about the Nonverbal Expression of Social Power. Journal of Nonverbal Behavior, pp. 105–123, 2005.
[8] A. Cafaro, H. Vilhjálmsson, T. Bickmore, D. Heylen, K. R. Jóhannsdóttir, and G. Valgarðsson, First Impressions: Users’ Judgments of Virtual Agents’ Personality and
Interpersonal Attitude in First Encounters. Intelligent Virtual Agents, pp. 67–80, Springer Berlin Heidelberg, 2012.
[9]M. Chindamo, J. Allwood, and E. Ahlsén, Some Suggestions for the Study of Stance in Communication. In Privacy, Security, Risk and Trust, 2012 International Conference on Social Computing, pp. 617-622. 2012.
[10] M. Chollet, M. Ochs, C. Pelachaud. Investigating Non-Verbal Behaviors Conveying Interpersonal Stances. First European Symposium on Multimodal Communication, 2013.
[11] B. M. DePaulo, Nonverbal behavior and self-presentation. Psychological Bulletin 111(2), pp 203–243 , 1992.
[12] A.H. Eagly and A. Mladinic, Gender Stereotypes and Attitudes Toward Women and Men. Personality and Social Psychology Bulletin, pp. 543-558, 1989.
[13] P. Ekman, W. V. Friesen, and Joseph C. Hager. Facial Action Coding System: The Manual on CD ROM. A Human Face, 2002.
[14] Q. Fu, R. op den Akker, M. Bruijnes, A Literature Review of Typical Behavior of Different Interpersonal Attitude, Capita Selecta HMI, University of Twente, 2014 [15] Q. Fu, R. op den Akker, M. Bruijnes, Typical Non-verbal Behavior of Different Interpersonal Stances during Police Interrogations, Research Topics report, HMI, University of Twente, 2014
[16] M. Gillies and D. Ballin, A Model of Interpersonal Attitude and Posture
41
[17] A. Garc´ıa-Rojas, F. Vexo, D. Thalmann, A. Raouzaiou, K. Karpouzis, S. Kollias, L. Moccozet and N. Magnenat-Thalmann, Emotional Face Expression Profiles Supported by Virtual Human Ontology. Computer Animation and Virtual Worlds, pp. 259-269, 2006.
[18] U. Hess and P. Thibault., Why the same expression may not mean the same when shown on different faces or seen by different people. Affective Information Processing. pp. 145–158. Springer London, 2009.
[19] M. L. Knapp and J.A. Hall, Nonverbal Communication in Human Interaction.Harcourt Brace College Publishers, 1972.
[20] D. Keltner, Signs of Appeasement: Evidence for the Distinct Displays of Embarrassment, Amusement, and Shame. Journal of Personality and Social Psychology, 68:pp.441–454, 1995.
[21] S. Kopp, L. Gesellensetter, N. C. Krämer, and I. Wachsmuth, A Conversational Agent as Museum Guide–Design and Evaluation of a Real-world
Application. Intelligent Virtual Agents, pp. 329-343. Springer Berlin Heidelberg, 2005. [22]R. Khan and A. De Angeli, The attractiveness stereotype in the evaluation of embodied conversational agents. In Human-Computer Interaction–INTERACT 2009 (pp. 85-97). Springer Berlin Heidelberg, 2009.
[23] N. Leßmann, A. Kranstedt, and I. Wachsmuth, Towards a Cognitively Motivated Processing of Turn-taking Signals for the Embodied Conversational Agent Max.
In Proceedings Workshop Embodied Conversational Agents: Balanced Perception and Action. IEEE Computer Society, pp. 57–64, 2004.
[24] J. M. Loomis, J. J. Blascovich, and A. C. Beall, Immersive Virtual Environment Technology as a Basic Research Tool in Psychology. Behavior Research Methods, Instruments, & Computers, 31(4), pp.557-564, 1999
[25] M. McRorie, I. Sneddon, G. McKeown, E. Bevacqua, E. de Sevin, and Pelachaud, C, Evaluation of Four Designed Virtual Agent Personalities. Affective Computing, IEEE Transactions on, 3(3), pp. 311-322, 2012
[26] C. Nass, Y. Moon and N. Green, Are Machines Gender Neutral?
Gender-stereotypic Responses to Computers With Voices. Journal of Applied Social Psychology, volume 27, Issue 10, pp. 864-876, 1997.
[27] L.P. Naumann, S. Vazire, P.J.Rentfrow, S.D.Gosling, Personality Judgments Based on Physical Appearance. Personality and Social Psychology Bulletin 35(12), pp. 1661–1671 , 2009.
[28] M. Neff, Y. Wang, R. Abbott, and M.Walker, Evaluating the Effect of Gesture and Language on Personality Perception in Conversational Agents, 10th International Conference, IVA 2010.
[29] J. A. Russell, J. Bachorowski, and J. Fernandez-Dols, Facial and Vocal Expressions of Emotion. Ann. Rev. Psychology, vol. 54, pp. 329-349, 2003.
[30] B. Ravenet, M. Ochs, and C. Pelachaud, A computational model of social attitude effects on the nonverbal behavior for a relational agent, Proc. of Workshop Affect Campaign on Artificial Interaction, 2012.
42
[31] B. Ravenet, M. Ochs, C. Pelachaud, From a User-created Corpus of Virtual Agent’s Non-verbal Behavior to a Computational Model of Interpersonal
Attitudes. Intelligent Virtual Agents, pp. 263-274, Springer Berlin Heidelberg, 2013 [32] J. Sanders, T. Smith, and J. Alexander, Type A Behavior and Marital Interaction: Hostile-Dominant Responses During Conflict. Journal of Behavioral Medicine, Volume 14, Issue 6, pp. 567-580, 1991.
[33]A. Schmeil and W. Broll, Mara: An Augmented Personal Assistant and Companion. In ACM SIGGRAPH 2006 Sketches, pp. 141, 2006.
[34] S. Spitters, M. Sanders, R. op den Akker, M. Bruijnes, The Recognition of Acted Interpersonal Stance in Police Interrogations, 4th IEEE International CogInfoCom 2013.
[35] A.Vinciarelli, M.Pantic and Herv´e Bourlard, Social Signal Processing: Survey of an Emerging Domain. Image and Vision Computing , 27(12), pp. 1743-1759 , 2008. [36] A. Vinciarelli, H. Salamin and M. Pantic, Social Signal Processing: Understanding social Interactions through Nonverbal Behavior Analysis. Computer vision and Pattern Recognition Workshops, pp 42-49, 2009.
[37] W. D. Wells and B. Siegel, Stereotyped Somatotypes, Psychological Reports, pp. 77-78, 1961.
[38] M. K. Young, G. B. Gaylor, S. M. Andrus, and B. Bodenheimer, A Comparison of Two Cost-differentiated Virtual Reality Systems for Perception and Action Tasks. In Proceedings of the ACM Symposium on Applied Perception , pp. 83-90, 2014 . [39] Z. Zeng, Pantic, M., G. I. Roisman, and T. S. Huang, A Survey of Affect Recognition Methods: Audio, Visual, and Spontaneous Expressions. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 31(1), pp. 39-58, 2009.
43
Appendices
Appendix A Glossary of Technical Terms
ArmatureAn "armature" is a type of object used for rigging. Armature object borrows many ideas from real life skeletons.
Rig
A rig is also a type of object used for rigging. It can be customized skeleton for any object. However armature only refers to human character rig.
Mesh
Mesh is also called polygon mesh. It is a is a collection of vertices, edges and faces that defines the shape of a polyhedral object in 3D computer graphics and solid modeling.
Rigging
After completing your character, you need to manipulate it for animation or just for posing. Rigging is the process of attaching a skeleton/armature to your character mesh object so you can deform and pose it in different ways.
Rigify
Rigify helps the process of rigging and automate the creation of character rigs. It is based around a building-blocks approach, where you build complete rigs out of smaller rig parts (e.g. arms, legs, spines, fingers). The rig parts are currently few in number, but as more rig parts are added to Rigify it should become more capable of rigging a large variety of characters and creatures.
Unity Mecanim
Unity has a rich and sophisticated animation system called Mecanim. Mecanim provides: easy workflow and setup of animations on humanoid characters, animation retargeting, Simplified workflow for aligning animation clips, convenient preview of animation clips, transitions and interactions between them, management of complex interactions between animations with a visual programming tool, and animating different body parts with different logic.
Blend Shapes
Blend Shapes, also called morph target animation, per-vertex animation, or shape interpolation, is a method of 3D computer animation. In a blend shape, a "deformed"
44
version of a mesh is stored as a series of vertex positions. In each key frame of an animation, the vertices are then interpolated between these stored positions.
Appendix B Blender Rigify to Unity Mecanim
import re import bpy
porg = re.compile('ORG-*')
for object in bpy.context.object.data.bones: object.use_deform = False
for object in bpy.context.object.data.bones: if porg.match(object.name): object.use_deform = True bpy.context.object.data.bones['ORG-heel.L'].use_deform = False bpy.context.object.data.bones['ORG-heel.02.L'].use_deform = False bpy.context.object.data.bones['ORG-heel.R'].use_deform = False bpy.context.object.data.bones['ORG-heel.02.R'].use_deform = False
45
Appendix C Facial Animation in Real-time
IEnumerator ChangePosture(int stateNameHash, int layerIndex, float target, float blendIn, float blendOut, float duration, float delay=0f)
{
float startTime = Time.time + delay; float holdTime = startTime + blendIn; float outTime = holdTime + duration; float totalTime = outTime + blendOut; float weight=0;
while (Time.time < startTime ) { yield return null;
}
anim.CrossFade(stateNameHash,0f,layerIndex,0.1f); while(Time.time < holdTime)
{
weight += target/blendIn * Time.deltaTime ; //speed of blend in anim.SetLayerWeight (layerIndex,weight);
yield return null; // wait for the next frame }
weight=target;
anim.SetLayerWeight (layerIndex,weight); while(Time.time < outTime){
yield return null; }
while(Time.time<totalTime){
weight -= target/blendOut * Time.deltaTime ;//speed of blend out anim.SetLayerWeight (layerIndex,weight);
yield return null; // wait for the next frame }
anim.SetLayerWeight (layerIndex, 0 );
46
Appendix D Body Action in Real-time
IEnumerator AnimateBlendShape(int blendShapeKey,float blendIn, float hold, float blendOut, float target, float delay = 0f)
{
float startTime = Time.time + delay; float holdTime = startTime + blendIn; float outTime = holdTime + hold; float totalTime = outTime + blendOut; float value = 0;
while (Time.time < startTime) { //delay yield return null;
}
while(Time.time < holdTime) {
value += target/blendIn * Time.deltaTime ; //speed of blend in sRenderer.SetBlendShapeWeight(blendShapeKey,value);
yield return null; // wait for the next frame }
value=target;
sRenderer.SetBlendShapeWeight(blendShapeKey,value); while(Time.time < outTime){
yield return null; }
while(Time.time < totalTime){
value -= target/blendOut * Time.deltaTime ;//speed of blend out sRenderer.SetBlendShapeWeight(blendShapeKey,value);
yield return null; // wait for the next frame }
sRenderer.SetBlendShapeWeight(blendShapeKey , 0); }
47
Appendix E Questionnaires in User Test I
Do you prefer to play the game with audio?strongly prefer audio prefer audio neutral prefer without audio strongly prefer without audio
The virtual suspect in the game is more realistic with audio. strongly agree agree neutral disagree very disagree
It is easier to judge the interpersonal attitude of the suspect with audio. strongly agree agree neutral disagree very disagree
I feel more concentrated when there is audio.
strongly agree agree neutral disagree very disagree
It is easier to judge the interpersonal attitude in the game setting than the video acted out by real actors. (Only applicable if you have joined the previous survey. If not, please skip this question)
strongly agree agree neutral disagree very disagree
It is more difficult to judge the interpersonal attitude of the virtual suspects when playing with Oculus.
strongly agree agree neutral disagree very disagree
Do you prefer to play the game with Oculus?
strongly prefer Oculus prefer Oculus neutral prefer without Oculus strongly prefer without Oculus
The suspect in the game is more vivid when playing with Oculus. strongly agree agree neutral disagree very disagree
The suspect in the game is easier to observe when playing with Oculus. strongly agree agree neutral disagree very disagree
48
It is easier to judge the interpersonal attitude of the virtual suspects when playing with Oculus.
strongly agree agree neutral disagree very disagree