Technical Factors

During AMC in an ICVE system, the properties of the visual representation of fellow interactants’ avatars is critical. These properties are defined by the technical characteristics of the ICVE system, both in terms of hardware and software. Regarding hardware, the brightness and resolution of the projection displays are central determinants, while for software, the fidelity of the geometric and articulated humanoid mod- els, together with graphical rendering quality, are the major impact factors. The combination of these features form the core of the visual stimuli on which the mediated social interaction unfolds. Hence, a low-luminance projection system, or a particular contour to the shape of an avatar’s head, are param- eters informing how a user experiences the AMC. This experience is critical to the ability and manner in which users may perform specific action. Although no absolute measures were taken, the CAVE’s projection system at UCL used by the experimental participants is relatively low-luminance compared to the system at UoR and UoS. As the central focus of this experiment was the investigation of varying methods of avatar gaze control, it should be noted that the visibility of the graphical representation of fellow avatars’ eyes, which appeared life-size (a small number of physical pixels depending on rendering perspective) was not optimal. Thus, it is conceivable that the nature of the CAVE display system used by the participant during the experimental interactions was detrimental to observation and subsequent utilisation of the visually-subtle nonverbal channel of gaze.

Similarly to the prior conversation experiment, and as discussed in Section 4.4, the same basic avatar model was used in the current experiment to represent all three participants (albeit displaying differently coloured shirts). While the model does feature photographic texturing, its geometric fidelity is relatively basic in comparison to the detail of a human face. Also likely to be a significant factor to an observer’s perception of the avatar was the lack of eyelid movement and blinking, causing the sclera of

5.5. Chapter Summary 153 the eye to appear prominent and unnatural. These graphical characteristics should again be considered non-optimal and, recalling Garau’s suggestion of there existing an interaction effect between visual and behavioural fidelity [Gar06], detrimental to the potential benefit of the faithful tracked gaze behaviour. Additionally, as exposed in the interaction analysis presented in Figure5.11, the lack of avatar mouth movement led to difficulties regarding interaction and conversation management.

The overall implication of these findings promote AMC in ICVEs as a platform able to support high- quality object-focused interaction. This is reinforced by the subjective judgements of the experimental experience gathered by the questionnaire and interview data. Ratings both of ease of collaboration and of involvement in the task were rated highly, for all gaze conditions, on the 1..7 Likert scale. As a final indicator of the intuitive nature of telecommunication able to be supported by ICVEs, it should be noted that, despite the use of na¨ıve participants and the complexity of the task, no critical failures of interaction occurred, and no outliers with regards to overall time taken were found.

5.5

Chapter Summary

The experiment reported in this chapter sought to investigate the impact of varying methods of avatar gaze control on quality of communication during object-focused multiparty AMC in ICVE systems. Through piloting, the experimental task, based on instructions issued by two confederates, and performed by a single participant, aimed to emphasise the operational importance of gaze during the collaborative interactions. The experiment compared three forms of AMC: tracked gaze, static gaze, and a simple gaze model. Data was collected from multiple sources, including eye tracking log files and videos, Eye- CVE log files, gaze model log files, and subjective questionnaires. This data was analysed as metrics of task performance, subjective user experience, and interaction analysis. Analysis of task performance and interaction analysis revealed that tracked and static gaze AMC is able to support superior quality of com- munication than modelled gaze AMC. However, measures of subjective user experience did not show a significant difference between the gaze conditions. While the original hypothesis that tracked gaze AMC would support a higher quality of communication than both static and modelled gaze AMC is not fully supported, participants’ action was significantly hindered in the modelled gaze condition, demon- strating the influence of misleading eye movement during object-focused AMC. Discussion focused on both experimental and technical characteristics of the studied AMC. Several factors, including display properties, avatar representation, na¨ıvity of participants, and task design, were suggested as likely de- tractors from participants’ ability and need to utilise, in practical terms, the richer nonverbal information supported by tracked gaze AMC.

The final telecommunication experiment presented in this thesis builds on lessons learned from Chapter4’s conversation experiment, and particularly, from the object-focused study documented in this chapter. Specifically, EyeCVE’s avatar subsystem is improved, increasing the visual and behavioural fidelity of user embodiments, and a bespoke semi-immersive VR system, featuring a high luminance and high resolution projection display, is utilised to both enhance visual clarity and to ensure robust tracking of additional oculesic cues of blinks and pupil dilation.

154

Chapter 6

Experiment: Truth and Deception

This chapter presents two closely related experiments investigating truthful and deceptive interaction in AMC. Similarly to the experiments presented in the previous two chapters, eye tracking is used both interactively and analytically. However, while the previous experiments only investigated gaze, the ex- periments in this chapter also investigate blinking and pupil size, which are captured, represented in avatars, and analysed. Thus, interactively, avatars reproduced their embodied user’s fuller oculesic be- haviour of gaze, blinking, and pupil size during AMC, and analytically, all three cues are examined. To the author’s knowledge, this is the first combined analysis of gaze, blinking and pupil dilation during social interaction in any field. The experiments are positioned in the social domain of interpersonal trust and deception, which presents a compelling array of issues by which to investigate interaction in visual telecommunication systems. Section2.1.3discussed how humans exhibit verbal and, critically, nonver- bal behavioural cues correlated with lying and truth telling, and how oculesic cues of gaze, blinking, and pupil size are influenced accordingly. Detection of deception was also covered, demonstrating that people estimate the veracity of a communicative message by observing others, so media transmitting fewer nonverbal channels becomes preferable for the deceiver [CGB+04].

The first experiment (E1) in this chapter explores truthful and deceptive discourse between dyads in state-of-art AMC and VMC systems. The experimental task design was inspired by Walczyk et al.’s question-answer framework for manipulating cognitive load when lying and truth telling [WSC+05]. During E1’s experimental interactions, a confederate issued questions to participants, who respond ei- ther truthfully or deceptively. Eye tracking was used to capture participants’ oculesic behaviour of gaze, blinks, and pupil size during both AMC and VMC for post-experimental analysis. Following the inter- actions, a questionnaire collected data describing participants’ psychological arousal and mood state.

The second experiment (E2) in this chapter investigates the impact of bestowing avatars with the oculesic cues of gaze, blinks, and pupil dilation, which are reproduced in real-time using eye trackers worn by embodied users. E2 follows on from E1, and invited a different set of participants to view audiovisual replays of E1’s experimental interactions. E2’s participants assessed E1’s participants’ de- grees of veracity and engagement, together with confidence levels relating to the two judgements. These ratings were performed over three stimuli conditions: avatars exhibiting oculesics, avatars featuring no oculesics, and audio-only replays.

6.1. Experimental Aims and Expectations 155

6.1

Experimental Aims and Expectations

The overall aim of the two experiments presented in this chapter was to investigate both behaviour (E1) and observation (E2) of truthful and deceptive discourse in AMC. The primary question E1 sought to answer is whether users’ behaviour and response during social interaction in AMC and VMC is sim- ilar. More specifically, when engaged in audio-visual telecommunication, if the visual component of the communication is depicted by the virtual, graphical, stimuli of AMC, will users exhibit nonverbal behaviour and psychological response that is similar to face-to-face interaction simulated by VMC? This aim appears similar to that of the experiment presented in Chapter4, which investigated behaviour in a multiparty conversational scenario in tracked gaze AMC and gaze aware VMC. However, that exper- iment was exploratory in nature, and rigorous analysis of oculesics was hindered by the ability of the MobileEye eye tracker. In contrast, E1 focuses on a specific social scenario (truth and deception) that is critical to practical usage of telecommunication systems, and considers a full range of oculesic be- haviour coupled with rich interaction analysis as presented in Section3.2. The comparison of AMC to the dominant form of visual telecommunication, VMC, aimed to provide a benchmark by which to measure users’ behaviour and response during interaction mediated by virtual embodiments.

E2 aimed to investigate avatar fidelity, and how the increased nonverbal richness afforded by oculesic reproduction of user behaviour may influence how judgements of AMC may be formed. Specif- ically, E2 sought to investigate whether observers of AMC are able to detect truth and deception more accurately when an avatar’s oculesic behaviour replicates the embodied user’s.

The experimental hypotheses (noted Hn) were as follows:

• E1H1: During AMC and VMC, participants will exhibit similar patterns of the oculesic cues of gaze, blink rate, and pupil dilation. However, psychological arousal, measured by questionnaire data, will be greater following VMC.

• E1H2: When communicating deceptive messages in both mediums: participants’ proportion of gaze directed at their partner will contrast to that measured when truth telling; participants’ blink rate will decrease, followed by compensatory blinks after speech has ended; participants’ pupils will dilate to a larger size than when truth telling.

• E2H1: When assessing the veracity of E1’s participants, judgements will be more accurate and more confident when observing avatars featuring reproduction of oculesic behaviour than judge- ments of avatars displaying no oculesic expression, or audio-only stimuli. Similarly, higher ratings of engagement, and confidence in this rating, will be elicited when observing avatars featuring oculesic behaviour.