Discussion

We now present several areas that the results of our study address, and details on how multimodal interactions relate to the attention-utility theme and how the results of the study validate the predictions of the framework.

Attention management. In the low interruption condition, where gestures were seen first, the number of primary tasks completed was significantly less for gestures than for the touch screen interaction. This suggests there is likely an element of difficulty in using gestures for the first time (most participants had little to no prior experience with gesture systems). This is also consistent with the reaction time for gestures being greater than the reaction time for the touch screen for the first set of trials. Based on observations, participants at first tended to use deliberate, slow gesture motions when reacting to notifications. However, by the second set of trials, participants had become accustomed to the interaction (i.e. accounting for warm-up time). During low interruption, we see a greater number of primary tasks completed in the second group of trials for gesture compared to touch screen. Semaphoric gestures provided a means of acting at a distance, which was useful for managing secondary tasks. When gesturing, users were able to maintain their focus the primary search task, and only had to glance at the secondary display before gesturing. In the touch condition, participants

Figure 6.6: Users ratings for interaction mode preference (Touch-gesture).

had to devote more of their attention to the secondary display because they physically had to move to the display and touch a specific area of the screen. Our results show the most benefit for gesture occurs when attention draw is low. The benefit appeared to be masked in the high interruption group, as there was no significant difference in performance. We note that the higher interruption was an increased rate of visual strobe with an accompanying sound; as expected, this was sufficient to reduce the reaction times. However, the greater level of distraction meant that participants lost the benefit of gesture over touch-screen interaction. We provided a spatially appropriate cue, which caused eye gaze to be diverted to the notification task. This effect interfered with the eyes-free benefit of gestures. While our results show that the significant differences in performance are primarily affected by condition rather than interaction mode, we observe that semaphoric gestures support less interruptive interactions with notification systems than touch-based interactions for low interruption secondary tasks. Our first hypothesis is thus supported for non-critical (less interruptive) secondary tasks.

Interaction mode and utility. Overall results suggest that reaction times and performance measures in the gesture condition do not differ significantly from the touch screen condition. This suggests that our second hypothesis is not supported when the interaction space for gestures and touch are similar. Since our interaction scenario is based on using similar interaction zones for both gestures and touch screen, we observe that the increased area of the interaction zone possible with gestures would thus be more suitable for a pervasive or ubiquitous computing environment where interactions at a distance may be appropriate. As the surface available to a system becomes larger, the interaction zone scales also, resulting in larger spaces for interacting compared to

Figure 6.7: Subjective results showing disruption to primary task organised by inter- ruption level conditions and mode.

what is typical of desktop applications. The performance benefits of semaphoric ges- tures over touch-based interactions would become clear as the interaction zone scales larger. The very low error rates in participant responses to notifications show that they had no problem understanding and responding to the notifications in both the touch and gesture conditions, showing that both are equally useful for performing relatively simple interactions that are often required from notification systems. However, if we also consider the subjective preference participants showed for the gestures over the touch screen, and overall primary task completion rates, we can deduce that gestures have the potential to improve utility (secondary task performance) while reducing the demands on visual attention.

Interaction mode and the attention-utility theme. We also varied the in- put mode in order to observe whether differential effects occurred in success rates of primary and secondary tasks. Results suggest that our third hypothesis is supported, as secondary task completion rates or comprehension levels were not effected by input mode, but by other factors such as familiarity with the task. Studying the impact of input mode on the user is an area that is just beginning to be explored in the design of notification systems. We selected touch screen as a conventional input in comparison to semaphoric gestures, a novel input method in this dual task scenario. Our hypothesis that semaphoric gestures can enable a more effective style of interaction that reduces distraction to a primary task was supported in the group with lower interruption. How- ever, during high interruption using gestures had no additional benefit compared to the conventional touch interaction. Given that the results show differential effects depending on level of interruption, we observe that in a multi-tasking situation the choice of input

mode can also have an impact on the user’s reaction to secondary tasks. In particular, gestures for responding to notifications permitted a lower level of distraction, whereas touch screen often resulted in loss of context in the focal search task. We note that the difference between gesture and touch disappeared in the high interruption group. The user response to a notification requires an allocation of attention, which may vary de- pending on how they interacted with the notification system. Thus, semaphoric gestures permit less interruptive interaction when distraction is already low; in terms of reaction and recovery times, the efficiency of reaction for gesture is similar. When considering these results, we can conclude that our fourth hypothesis; that gestures are more efficient for managing attention-utility trade-offs is supported for non-critical notification tasks.

Interaction mode and user preference. Since gesture currently is a novel input mode for many users, this type of interaction can require a period of adjustment. This suggests that semaphoric gesture is not as natural an interaction technique and thus does not engage our preexisting cognitive and physical resources as effectively. How- ever, we observed that gesture performed as well as touch-screen, and was subjectively the preferred mode of interaction for the majority of users. This shows that our fifth hypothesis that semaphoric gestures are as intuitive and easy to use as touch-screens to interact with notifications is only partially supported. Most participants experienced gestures as a less disruptive interaction, and it permitted easier resumption of their primary task. For gesture, several users reported an ability to search without feeling tied to the secondary display. The benefit of gesturing in-place avoids re-purposing ones hands to carry out a secondary task such as responding to a notification, and avoids changing ones physical position such that the current context in the focal search task is lost. Thus, after a period of adjustment, gesturing was seen as easier by the majority of participants.

Notification systems and multimodal interactions. Using the IRC framework, we were able to select two different styles of notification, knowing in advance there was a salient difference in the level of interruption. This allowed us to expect certain behaviour based on our design decision. In particular, we expected faster reaction times for higher interruption, without any impact on the performance in the primary task. We also expected gesture to be a better means of interaction in all cases based on the following design claim: Semaphoric gestures for secondary tasks are convenient for action at a distance and provides opportunity for eyes-free responses to notification (i.e. use of peripheral vision) but may require more time to execute compared to traditional input methods. In this study, varying the level of interruption allowed us to probe two areas of the IRC design space. Our results highlight the importance of addressing different input modalities and the usefulness of the IRC framework for designing notification systems. First, the level of interruption has a direct influence on the user’s reaction as expected. Second, the input mode has an impact on reaction that must be considered

choice of input mode will favour semaphoric gesture when lower distraction is desired; this will be made possible by supporting the user’s ability to act at a distance. Our work shows how design goals, stated here in terms of the attention-utility theme and the IRC framework, can be related to effective choice of input modality for a specific class of ubiquitous systems.