One of the key features we claim distinguishes gesture interactions is the different sys- tem responses or output produced using gestures. In our review of the literature, we noted that gestures, as with any input, results in an output. The result of a gesture is presented in some form to the user, and represents a major factor in the user experience. For our literature review, we approach the modality of a system response in terms of the resulting audio or visual (2D and 3D) display output, or as simply the resulting pro- cessor command (CPU). We next present system responses discussed in the literature, summarised in Figure 2.7.
Visual Output. Most system responses for gestures result in a visual display output, however this is mainly controlled based on the nature of the interaction. For example, manipulations of graphical objects on a visual display would provide persistent results of transformation on that same display. However some gesture interactions do not re- quire visual feedback but rather, produce audio output or other types of commands for which users may not require a visual response. Visual output in on 2D displays are
Figure 2.7: The bar graphs shows the distribution of system responses through output
technology used in the gesture research reviewed in this paper.
most commonly related to visual interactions with desktop screens, large screen dis- plays, projected displays and portable or mobile devices including phones and PDAs. While output is often implied by the scenario —on-screen interactions result in on-screen responses —unexpected responses are also common in gesture interactions, where an on- screen interaction may result in audio output (Schmandt et al., 2002; Pirhonen et al.,
2002), or simply invoke CPU behaviour in other devices (Cao & Balakrishnan, 2003). Similarly, 3D visual output using head mounted or stereoscopic displays produce output to communicate the effects of an intended gesture to the user through the appropriate perceptual channel. The main point we wish to make in this category is that while input device can change, output devices can also change, and each must be considered indi- vidually to fully understand the nature of the interaction. For example, within virtual reality applications, the display can be immersed, semi-immersed or non-immersed, each with specific properties that can influence how we gesture. For example, fully-immersed displays may provide affordances for 3D gestures using glove inputNishino et al.(1997);
Song et al.(2000), while non-immersed 3D projected displays may be more conducive to bare-hand gestures Sharma et al.(1996). 3D graphics can be presented on 2D displays (Wexelblat, 1994; Nishino et al., 1997) where glove-based manipulative gestures seem
natural, and mouse gestures may not. Other visual output such as the volumetric dis- play (Grossman et al.,2004) and 3D projection systems (Sharma et al.,1996) may also afford hand gesture interactions, however we maintain that understanding the output technology will better lead the design of suitable gestures.
Audio output. Research on gestures that lead to audio output demonstrates how a simple task can be executed and provide responses without requiring visual attention (e.g. Schmandt et al., 2002; Pirhonen et al., 2002; Brewster et al., 2003). While most audio output used in gesture research is situated around mobile and pervasive comput- ing, the main function for non-speech audio is feedback, which is applicable to most computing domains. We investigate different types of audio response or feedback used with gesture interactions in Chapter6.
CPU: command directed responses. Often, the response of a recognised gesture is not required to be directed to a specific output device. We refer to this as a CPU response, where the resulting output of a gesture is simply stored, or redirected to another device. For example, pointing gestures that identify devices in a smart room may lead to shutting down a device, or changing the state of the device, and may not lead to additional system responsesWilson & Shafer(2003). The responses can also lead to multiple responses as determined by the system (e.g. Pausch & Williams, 1990;Roy et al.,1994;Keates & Robinson,1998;Reilly,1998), or can lead to a variety of responses from different devices within smart room or ubiquitous environments (e.g.Gandy et al.,
2000;Fails & Olsen, 2002;Wilson & Shafer,2003;Nickel & Stiefelhagen,2003).
2.5.1 Analysis of System Responses
The primary response of a gesture interaction remains directed to desktop style, 2D visual displays. However, as ubiquitous computing domains are explored, we should see a move away from the more traditional output, towards more audio and CPU responses. One trend we note is that novel output technologies often become targets of gesture research. There is a need to move away from direct input devices to seek more natural and novel interactions, and gestures can potentially fill that gap. Volumetric displays (Grossman et al.,2004) and 3D projections (Sharma et al.,1996) present opportunities for employing manipulative gestures to control the display output. However, the rela- tionship between these novel outputs and gestures are tightly bound by the nature of the task and the enabling technologies that are associated with a given interactive sys- tem. Often these novel technologies are on the cutting edge, where accessing expensive devices make it difficult to conduct significant user studies. We discuss the different types of evaluations conducted on gesture research next.
Figure 2.8: Evaluations of gesture research within Application Domains.