Emergent Gaze Behaviour

In both the AMC and VMC, participant gaze behaviour was similar, and comparable to collocated in- teraction. Participants did not solely look away from confederates, nor gaze fixedly at them, and gaze was dependent on the current state of the ongoing interaction. However, a particular longitudinal phe- nomenon was observed in the AMC sessions with regards to how participants utilised gaze as the ex- periment progressed. During the early minutes of interaction, it was observed that two of the five AMC participants directed their gaze generally towards the confederate avatars, as opposed to focusing on the face or attempting to establish mutual gaze. However, as the interview progressed, these participants began to consistently target confederate avatars’ eyes. This emergent behaviour is illustrated in Fig- ure4.15, and may be explained by participant’s initial inexperience of system, which, as it increased, led to the realisation of the relevance of other avatars’ gaze behaviour, and subsequently elicited more natural gaze behaviour from the participant themselves. Participants had no previous experience of im- mersive VR, and hence, the novelty of the scenario of interacting with remote people via life-size avatar embodiments was likely to elicit different attentional and gaze behaviour during acclimatisation to the medium. From analysis of eye tracking videos of the two participants who displayed the emergent gaze behaviour, the change from general targeting of confederate avatars to specific gaze directed at the eyes and face occurred towards the end of the first minute of the interview, as marked in Figure4.15. This, more socially normal gaze behaviour, particularly of attempting to establish mutual gaze, remained in practice for the remainder of the interactions, and was comparable to the other three AMC participants, who demonstrated such gaze strategy throughout.

As discussed in Section4.2.2, the VMC setup allowed only head and shoulders views of confed- erates. Hence, when looking towards confederates, the potential for participants to look anywhere apart from their faces was limited in comparison to the full-body and unified environment presented during

4.5. Chapter Summary 129 AMC. Hence, while direct comparison of longitudinal trends in gaze behaviour between AMC and VMC would not be valid, all VMC participants sought to establish mutual gaze with confederates, throughout the entire course of the interactions. This was implied quantitatively in the gaze distribution analysis in Section4.3.3, which demonstrated significantly higher proportion of gaze directed at the questioning confederate during VMC than during AMC. Combined with observations of emergent gaze behaviour in AMC, this suggests that participants fostered a higher degree of social presence when confederates were represented by live video than by avatar embodiments. The limitations of the basic avatars used in this experiment are likely to have influenced participant gaze behaviour during AMC. In particular, a lack of mouth movement, facial expression, and eyelid movement were likely to be detrimental to the social realism of the embodied virtual humanoids.

4.5

Chapter Summary

The experiment presented in this chapter aimed to compare the behaviour of participants engaged in a triadic conversational scenario performed using tracked gaze AMC and gaze aware VMC. An early version of EyeCVE supported the AMC, while AG nodes at the three interacting sites were setup as optimally as possible to enable gaze awareness between interactants. A semi-structured role-play was formulated, featuring two confederates interviewing a single participant. Qualitative CA combined with eye tracking indicated that interaction in both mediums is influenced by technical characteristics of the system, particularly with regard to workspace fragmentation and user representation. However, interac- tants were seen to adopt strategies to successfully manage conversational turn-taking in both mediation types: in VMC, confederates generally employed gaze and facial expression to signal transfer of the conversational floor, while in AMC, this was usually handled through hand gesture combined with gaze and head orientation.

The findings support the hypotheses that multi-party conversation is able to be conducted ade- quately, and that gaze behaviour will be employed similarly during both AMC and VMC for purposes of both nonverbal expression and management of the unfolding interaction. Gaze behaviour in both telecommunication mediums approximately followed established social norms observed in collocated in- teraction. This experiment was purposefully less formal, in terms of both design and analysis, and more exploratory, in terms of the general issues regarding gaze-enabled telecommunication systems, than the experiments detailed in the forthcoming two chapters. Discussion made both general and specific ob- servations regarding the advantages and deficiencies of the two mediums. In particular, limitations of EyeCVE’s avatar system were uncovered, and are subsequently developed and explored over the follow- ing chapters. Regarding data analysis, a critical limiting factor in this experiment was the MobileEye eye tracker’s low tracking robustness and the lack of detailed logging ability. The units are replaced by the ViewPoint eye tracker in the remaining work presented in this thesis.

130

Chapter 5

Experiment: Object-Focused Scenario

This chapter presents an experiment designed to evaluate the impact of varying methods of avatar gaze control in a three-party object-focused collaborative task. The experimental design takes Schroeder et al.’s [SSA+01] simplified Rubik’s cube study as a primary influence. In that experiment, performance and subjective experience was compared over varying technologies (reality, immersive AMC, and desk- top AMC) in a collaborative task performed by two na¨ıve participants. The participants worked together to solve the 3D puzzle, and no structure was imposed on the interaction, which progressed based upon the action of, and communication between, the two participants. Accordingly, in their post-experimental analysis of the AMC, Schroeder and colleagues sought to explore their paper’s titular question: is col- laboration in ICVEs as good as being there together?

In contrast, the experiment presented in this chapter imposes a structure on the experimental AMC, defining a sequence of interactional subtasks performed between two confederates and single participant. Informed by initial piloting of an object-focused scenario, the structured interaction was designed to emphasise the communicative importance of gaze as an interactional resource in AMC, and also to enable rich analysis of the recorded data. Hence, while Schroeder and colleagues were concerned with how varying technologies may influence collaboration, the current experiment focuses on the impact of varying methods of avatar gaze control on quality of communication in immersive AMC. The goal of the experiment was to compare tracked gaze, a simple gaze model, and no eye movement during an object- focused task. The methods of analysis are extended from those employed in Chapter4, and include task performance, subjective user experience, and interaction analysis using CA coupled with eye tracking data.

5.1

Experimental Aims and Expectations

The aim of this experiment was to assess the benefit of tracked gaze AMC in a multiparty task that required physical movement and object manipulation. Gaze aware VMC systems preclude users’ abil- ity to move freely within their local environment while maintaining meaningful communicational gaze with remote others. Hence, in contrast to the experiments presented in the preceding and subsequent chapters, the current experiment does not investigate VMC, and instead focuses on varying conditions of avatar gaze control in immersive AMC. The ability to both manipulate and share virtual objects in a

5.2. Experimental Design 131

Figure 5.1: Views of the completed ‘Rubik’s cube’ puzzle, formed by arranging eight small cubes to form one large cube in which each side displays exactly one colour.

unified shared workspace is a feature unique to ICVEs, and forms an integral part of the experimental interactions. The overall goal of this experiment was to investigate differences between varying methods of avatar gaze control, as measures of task performance, subjective user experience, and interactional behaviour.

The primary hypothesis was that tracked gaze AMC would be able to support a higher quality of communication, measured by task performance, subjective user experience, and interaction analysis, than both static and modelled gaze AMC. Specifically, when all interactants are embodied by tracked gaze avatars, this form of AMC is expected to outperform the two other classes of avatar gaze control in the following ways: action will be performed faster and with fewer errors; participants will rate overall communication as more natural, and instructions issued by interactional partners as less ambiguous; dur- ing interaction, participants will demonstrate gaze practices that are more comparable to those observed during collaboration in the real-world; and finally, when a mistake is made during the collaborative task, participants will be able to more effectively employ repair strategies to remedy the situation.

5.2

Experimental Design

A within-groups repeated measures design was used, with twelve participants performing three consec- utive experimental sessions in each of the three avatar gaze conditions of tracked gaze, simulated gaze, and static gaze. Similarly to the previous conversation experiment, participants took part in three-party interactions comprising of themselves and two confederates: one located at the University of Reading (UoR) and the other at the University of Salford (UoS). The same two male confederates performed interactions with all twelve male participants. The main component of interaction involves the two con- federates alternately issuing instructions to the participant, who performs the requested action. Once the instruction has been completed successfully, the next instruction may be issued. This process continues until the entire sequence of instructions has been issued, and the goal state of the puzzle is achieved; at which point the current experimental interaction concludes. Figure5.1shows the puzzle’s goal state, rendered from various angles, illustrating the arrangement of eight smaller cubes to form a single, larger cube, with each side displaying exactly one colour.

The core of the collaboration is the interactional sequence of confederates issuing instructions to a participant, and the participant performing the requested action. Each experimental session comprises

5.2. Experimental Design 132 a total of ten instructions toward achieving the goal state of the puzzle. These ten instructions are logically paired, resulting in five pairs per experimental session. Each instruction pair comprises one grabinstruction, and a follow-on position instruction, both of which are issued by the same confederate. Grab instructions request the participant to take action to identify and pick up a specific cube being indicated by the instructing confederate. Subsequently, position instructions require the participant to identify and place the grabbed cube in the specific position, as indicated by the instructing confederate, within the cube puzzle. Following the participant’s successful grab and position action, the bystanding confederate (i.e. the confederate who did not issue the previous instruction) assumes control of the floor, and issues the next instruction pair. This process continues until all five instruction pairs have been issued, and the goal state of the puzzle is achieved.

In summary, this study does not present a symmetrical collaborative task as seen in Schroeder et al.’s experimental design [SSA+01]. Rather, the use and importance of gaze as a bidirectional resource in AMC, for both indication and observation of attention and action, are investigated over three varying methods of avatar gaze control. The following sections detail the experimental design. For additional materials, see AppendixD.

5.2.1

Independent Variables

The three methods of avatar gaze control are listed below in ascending order of their expected contribu- tion to quality of communication. Participants performed a total of three experimental sessions (one in each condition), in which both confederate avatars and their own avatar’s gaze was controlled by identi- cal methods described by the conditions. Hence, the following independent variables are termed static gaze AMC, modelled gaze AMC, and tracked gaze AMC. The order of conditions were resequenced over the twelve participants to negate the effect of learning and experience. Additionally, three VEs with different starting configurations were also built, and presented to all participants in the same order. The VE design is described in Section5.2.3.

1. Static gaze AMC: In this condition, avatars’ eyes are centred and do not exhibit any movement. This is the most basic form of AMC investigated in the experiment, and aims to provide a baseline from which the latter two conditions may be judged. When operating with static gaze, the attention of others must be inferred from alternative, tracked, nonverbal cues such as head orientation and gesture.

2. Modelled gaze AMC: In this condition, a basic simulation of gaze behaviour controls avatars’ eye movement. The characteristics of the model are described in Section5.2.3. A central feature of modelled gaze avatars is that they display gaze behaviour that is different to their embodied user’s actual eye movement.

3. Tracked gaze AMC: In this form of AMC, avatars’ gaze mimics that of their controlling users. Thus, users engaged in tracked gaze AMC will be able to both signal using their own gaze, and observe others’ gaze behaviour. The head-mounted eye tracking setup used to achieve this high- fidelity form of AMC is outlined in Section5.2.3.

5.2. Experimental Design 133

5.2.2

Piloting

Prior to running the experiment as it is documented in this chapter, the experimental design was refined over several iterations. The experiment aimed to assess the benefit of tracked gaze AMC in a multi- party task requiring physical movement and object manipulation. Initially, pilot sessions involving a three-party version of Schroeder et al.’s [SSA+01] symmetrical collaborative task, were carried out, in- vestigating the same independent variables of gaze control described above in Section5.2.1. However, while thorough interaction analysis would have likely uncovered subtle differences in communicative behaviour, it quickly transpired that task performance as a whole seemed relatively unaffected by avatar gaze condition when users were engaged in such unstructured interaction. Two main reasons may ex- plain this finding: the puzzle task is extremely involving, and while the three participants often verbally communicated, the success of the object-centric action did not necessarily require those involved to pay close attention to the visual presence of their fellow collaborators. This was compounded by the approx- imately hip-level height of the puzzle base (see Figure5.1) displayed in the CAVE systems, leading to participants generally adopting a downward-gazing posture. Secondly, three collaborators appeared to be one too many for the particular task, and one or two members generally emerged as dominant, thus reducing the cooperative nature of the task. A subsequent version of the experiment was also trialled, with two confederates acting as aides to a single participant, suggesting and helping them toward solving the puzzle. This design also suffered from the aforementioned problems.

Experience with pilot experiments suggested that, in order to investigate the operational benefit of using tracked gaze avatars in object-focused scenarios, the experimental task must emphasise the communicational importance of gaze during the collaborative interactions. Preliminary analysis of pilot experiments also suggested that some form of structure should be imposed on the interactions in an attempt to minimise the intervening variable of participant ability and approach, and also to enable data collection of specific and repeated sequences of action. The paradigm of an instructional scenario was developed, transforming the initial symmetrical and open-ended collaborative puzzle solving into a series of sub-tasks acting as components of a whole, and which featured the same eventual goal state. Pilot studies involving a single participant being instructed by two confederates were conducted successfully, and evolved into the final experimental design as described in Section5.2.5. The role of confederates was established as a key element of interaction, serving multiple purposes: the communicative importance of fellow users’ visual presence was emphasised; participant behaviour was controlled and constrained by a defined sequence of instructions; and the time taken from beginning to goal state of the puzzle was significantly reduced by the instructional design and expertise of confederates. From the practical perspective of experimental duration, which dictated the amount of time participants would be required to spend within the CAVE, the redesigned task enabled a within-subjects design in which participants performed experimental sessions in each of the three avatar gaze conditions. Finally, and as explained in Section5.2.3, the difficulty of the task was increased by the addition of several “dummy” cubes, alongside the “puzzle” cubes, into the VEs. The dummy cubes acted as decoys for the participant when carrying out grab instructions issued by confederates.

5.2. Experimental Design 134

5.2.3

Apparatus

EyeCVE supported the object-focused AMC between the three CAVE laboratories located at UCL, UoR, and UoS. At all sites, ViewPoint eye trackers were fitted to the local user, recording gaze. The following sections present the arrangement of the technical apparatus.

EyeCVE Setup

The EyeCVE setup for this experiment was similar to that described in Section4.2.2. A more mature version of EyeCVE was used, featuring improved network performance, and, due to the integration of the ViewPoint eye trackers, more frequent gaze updates to the avatar subsystem when tracked gaze AMC was in operation. The basic avatars documented in Section3.1.4were used, capable of displaying the oculesic behaviour of gaze, but not supporting eyelid kinematics or changes in pupil size. Similar to the previous conversation experiment, EyeCVE ran on SGI PrismTM_{computers at UoS and UoR. However,} UCL migrated to a Microsoft Windows R_{XP cluster, which operated with identical display properties:} 1024×768 pixels per wall, projected at 96-120 Hz in active stereo. Head and hand tracking remained identical at all sites to that described in Section4.2.2. UoS ran the centralised server, which maintained the state of the shared VE and distributed changes to all clients. Wireless microphones were worn by all interactants, and audio communication was achieved using a Skype [Sky10] conference call, and output through speakers located in each CAVE.

Three VEs featuring different starting configurations for the three experimental sessions were built. As the independent variables were resequenced over the twelve participants, the presentation order of the VEs remained static in order to negate the impact of any variation in difficulty of starting configuration. The spatial volume of the VEs was approximately equal to that of the CAVE systems (3×x3×2.2 m), allowing participants and confederates to navigate to any position within the VE naturally, through phys- ical movement, as opposed to requiring the use of an additional input device. Each VE was populated with a configuration of eight puzzle cubes and five dummy cubes. The puzzle cubes were coloured in order to have specific positions within the overall 2×2×2 goal state, while the dummy cubes appeared to be similarly coloured, but would not fit correctly into the finished puzzle. The purpose of the dummy