Users Goals

Our work considers user goals and the attention-utility theme, where the multitasking situation requires that users maintain focus on a primary task, while ensuring that secondary tasks are correctly completed.

6.3.3 Experiment

We created a dual-task situation to simulate a command and control environment within the interaction model applied to this experiment. The primary task was a search task,

Approach. We investigated different settings of IRC values for notification system interactions to test how input modes can effect users and to determine their role in attention-utility trade-offs for designing multimodal command and control environments. We made the following hypotheses:

1. Interruption: Semaphoric gestures would support less interruptive interactions with notification systems, resulting in higher primary task performance.

2. Reaction: Semaphoric gestures would support more efficient reaction to both high and low interruption notifications, resulting in higher secondary task performance. 3. Comprehension: Users would experience similar comprehension levels across the different notification systems using either touch or gesture interactions resulting in similar success rates for secondary tasks.

4. User satisfaction: Semaphoric gestures would be as intuitive and easy to use as touch-screens to interact with notifications, resulting in higher subjective ratings. 5. Efficiency: Semaphoric gestures could better optimise attention-utility trade-offs,

thus making them the preferred interaction mode for their perceived benefits.

We simulated user interactions on the peripheral display using the Wizard of Oz (WoZ) methodology, and provided two levels of interruption, low and high. The wizard used a wireless keyboard to control the peripheral display based on the participant’s actions (gesture or touch). We chose the WoZ methodology rather than a working gesture recognition system to prevent confounding data resulting from recognition errors. Par- ticipants used single right-handed gestures or were required to touch the required area of the screen for the secondary task interaction. Participants were free to use either hand for the primary search task.

6.3.4 Experimental Design

The experiment was a full factorial, mixed-model, repeated measures design with two interaction modes (gesture and touch) as the within-participant factor, and notification system interruption level (high or low) as the between-participant factor. The interaction modes were counter-balanced within participants. For the low interruption condition,

Figure 6.2: Overhead view of experiment layout with [c]amera, [p]articipant, [w]izard,

kb = keyboard.

Figure 6.3: A user engaged in focal search task (left) in the presence of a secondary

notification task (right).

visual notification was used, and in the high interruption condition, both an audio and visual notification were presented to the participants. We tested 4 conditions, coded according to the interruption level of the notification (0=low, 1=high) and counterbal- ancing (G=gestures seen first, T=Touch screen seen first). The four conditions are thus 0G, 0T, 1G, and 1T. In this paper, we define interaction zone as the area in which a user can conduct purposeful interactions with the system; as shown in Figure 6.2. For the gestures, the interaction zone coincides with the camera’s field of view, and with the touch screen, the interaction zone is the area on the screen that the user must touch to issue the command.

near the secondary display for a spatially appropriate cue. Figure 6.3 shows a partici- pant working on the primary task during the experiment. Our prototypical gesture set consisted of left-ward, vertical, and right-ward hand motions. We chose gestures that were easy to perform and had straightforward mapping to concepts (left, middle, right). We wished to minimise the cognitive effort required to perform the correct semaphore in response to notifications and to ensure that the touch and gesture interactions posed sim- ilar physical requirements. In this case, the gesture path encoded the relative positions of the coloured bars present in the notification display. The gestures were understood to serve the function of acknowledgement as well as having specificity for the particular notification issued. We define our dependent and independent variables next.

Independent variables. The independent variable tested for the within-participant condition was interaction mode. Each participant used both the touch screen and the gesture interaction. The between-participant variable was the notification system inter- ruption levels for secondary tasks: Low interruption provided visual only notifications and high interruption used both visual and audio notifications.

Dependent variables. To determine the measurements for participant reaction, comprehension, the degree of interruption to the primary task and the efficiency of semaphoric gesture vs. touch screen, we measured reaction time (reaction), success rate for responding to notifications (comprehension), the time taken to recover from the interruption to the primary task (recovery) and secondary and primary task times (efficiency), defined below. We also gathered subjective data on user preference and satisfaction ratings for each interaction mode using a post-evaluation questionnaire. Our experiment software recorded the times for each notification issued and the time that the primary tasks were resumed after completing the secondary task. Response times for notifications were logged using a key-press at the Wizard’s station. We measured the relative ease of execution of gesture vs. touch based on significant differences in reaction and recovery times. To infer effects on attention, we calculated the search success rates and measured the search times (primary task). We also recorded the number of secondary tasks completed and the total time to perform them. A detailed definition of the dependent variables is given below:

• Reaction: The time taken to respond to a notification, measured from when the notification is issued to when the user reacts by touching or gesturing.

Table 6.2: Mean times for reaction and recovery for gestures and touch screen.

• Recovery: The time taken for the participant to resume the primary task, measured from when the secondary task ends and the primary task resumes, representing the level of interruption caused to the primary task.

• Primary task success rate: The number of times an image was found during the focal search task.

• Secondary task success rate (comprehension): The number of times the partici- pants were able to correctly respond to the alert within the allocated time frame.

• Primary and secondary task times: The total amount of time participants spent during the search tasks, and responding to the notifications.

6.3.5 Results

To check for significant differences in the results, we first ran a mixed-model repeated- measures analysis of variance (ANOVA) to test the effects of interaction mode within participants and condition (order in which the two modes were presented to participants and the interruption level) between-participants. A second multivariate analysis of vari- ance (MANOVA) was run to examine the effects due to the independent variables of mode (gesture or touch), interruption level and counterbalancing in isolation of the four conditions. While our ANOVA showed no significant results for the within-participants factor of interaction mode, significant results were found in the differences between par- ticipants for primary and secondary tasks success rates, reaction time and secondary task time for the four conditions, discussed next.

Reaction and recovery time. Reaction time was significant in the ANOVA, with the mean reaction time for gestures faster in all but the 0T condition. This may be due to the more relaxed nature of the interactions within the low-interruption level condi- tions and possibly due to dealing with the novelty of the gestures in the second set of trials. The MANOVA revealed that an interaction effect existed between modes and

Figure 6.4: Estimated marginal means of recovery time for modes: gesture and touch. Recovery time when gestures were shown to participants first were higher. This is likely due to the novelty of the gestures. We see lower recover times when gestures are seen in the second set of trials indicating that there is a learning effect present for the trials.

counterbalancing, thus suggesting a learning effect was present. The MANOVA also revealed that interruption level yields a faster reaction time in a high interruption no- tification than for low interruption (F(1,32)=6.383, p<.05). While no interaction effect

was shown between gestures and interruption level in the MANOVA, reaction times for gestures tended to be faster than those for touch screen (see Table6.2). An interaction effect is shown for reaction time, with mode and order of presentation, suggesting that a learning effect positively affected reaction times for gestures in certain cases: In con- dition 0G, reaction times for touch actually increased, while in 1T reaction times for gesture decreased during the second trial, with 0T and 1G having similar reaction times (F(1,32)=9.583, p<.005) (see Figure 6.4). Our analysis did not show any significant dif-

ferences in the recovery times, however, there is a definite trend towards faster recovery using gestures over the touch interaction, as seen in Table 6.2. This applies to all but the 1G condition, where gestures appear to lead to a slower recovery time than for the touch interaction. While the slower recovery time may be due to increased interruption level, and to the novelty of using gestures, the differences are not shown to be significant in this model.

Primary and secondary tasks. Significant results for interaction mode in the ANOVA for primary tasks in the 0G condition with the gesture interactions yielded a greater number of primary tasks completed (F_(1,8) =6.733, p<.05). Secondary tasks were also significant in this model (F(1,8) =5.829, p<.05), however there were fewer

Table 6.3: Mean values for primary and secondary tasks completed for the two inter- action modes used in the experiment.

in the 0T condition, however gestures lead to lower success rates for primary tasks (F_(1,8) =6.897, p<.05) but a greater success rate for secondary tasks (F_(1,8) =9.218, p<.05). The 1T condition suggest that gestures also support a higher success rate for primary tasks (F(1,8) =9.529, p<.05). No significant results were found for the 1G

condition. The MANOVA results show that overall, completion of secondary tasks are significantly greater in the second set of trials (F(1,32) =14.286, p<.001) which may

explain the difference in primary task success rates between the four conditions in terms of mode from the ANOVA. There is also a significant interaction effect present for primary task success rate (F(1,32)=16.711, p<.001) due to the factors interruption level

and presentation order. In the first set of trials, there are more primary tasks completed for low interruption than for high interruption. However in the second set of trials, fewer primary tasks were completed in the high interruption condition than for low interruption. This suggests performance degradation occurred with higher interruption since it draws attention away from the primary task.

Subjective results. Despite the limited differences observed for the performance measures, participants showed an overall preference for gestures over the touch screen interaction (mean 7.15/10 for gesture). Participants rated gestures slightly higher than touch-screen for ease of interaction (mean of gestures=7.41; touch=7.22) but perfor- mance was rated as slightly lower (mean of gestures=8.52; touch=8.78). Ratings for ease of resuming the primary task were higher for gestures (mean gesture=7.11; touch=5.22), and lower for attention required for secondary task (mean gesture=5.37; touch=6.74), and distraction caused (mean gesture=4.67; touch=6.63), shown in Figure 6.5. When we examine these results, participant perceptions were that gestures required less atten- tion in the low interruption condition than in the high interruption condition. Touch screen interaction was roughly equivalent for both high and low interruption conditions. Gesture and touch were both rated as less disruptive in the high interruption condition while resuming tasks in general was rated as easier in the low interruption condition.

Figure 6.5: Summary of subjective ratings for interaction mode, gesture or touch screen.