This dissertation is structured to emphasize the contributions in the three main areas of research we focused on that are relevant to the topic of sensor-based mobile interaction: continuous interaction us- ing model-driven user interfaces, sensor-based and around-device in- teraction and, lastly, motion gestures. In the following, we outline the further structure of this dissertation.
In Chapter 2, we survey related work in order to obtain an overview of the state of the art in sensor-based mobile interaction. We highlight how the contributions in this dissertation build upon or extend related work in the eld.
We present the results of our research on continuous interaction with model-driven user interfaces in Chapter 3. This chapter portrays how the sensors embedded in current mobile devices can be used to improve the usability of mobile user interfaces by allowing the development of user interface models that simultaneously control a larger range of user interface parameters than the user would be capable of doing manually. In addition, the results in this section support our argument that by us- ing the built-in sensors of mobile devices more effectively, the usability of mobile user interfaces could be signicantly improved in the future. We examine future sensor technologies that could be incorporated into mobile devices in Chapter 4. We introduce the concept of around- device interaction (ADI) as a way of extending the input capabilities of mobile devices to encompass their entire surface as well as the phys- ical space surrounding them. To evaluate the concept ADI, we describe three hardware prototypes, and present the results of user studies ex- emplifying the usability improvements for devices supporting ADI in comparison to conventional mobile user interfaces.
1.3 Dissertation Structure 17
Motion gestures are a promising user interface input technique for mo- bile interaction. A difficulty for developers intending to make use of motion gestures in their applications is that in order to recognize non- trivial motion gestures, developers may need knowledge of machine learning techniques and need to resort to specialized libraries or toolk- its. One of the goals of this dissertation is to facilite access to mobile gesture recognition for practitioners and developers. In Chapter 5 we thus present a set gesture recognizers that feature a low implementa- tion effort and are easy to incorporate into mobile applications. The algorithms were developed to support rapid application development and prototyping. In this chapter we also cover a novel application for motion gestures, gesture-based authentication (GBA). Through the re- sults of a user study we show this type of authentication is both feasible to implement on mobile devices and resistant to visual attack attempts. The chapter concludes by an analysis of the effects of combining ac- celerometer data and gyroscope data on the accuracy of motion gesture recognition.
With the goal of making gestural interaction available to the end user and supporting prototyping of mobile gestural interfaces, we present Mayhem in Chapter 6. Mayhem is an end-user programming environ- ment that can be used to prototype gestural interactions for scripting tasks or home automation. We present a sample application case high- lighting Mayhem’s salient features.
In Chapter 7, we address open issues and enumerate possibilities for future continuation of the work presented in this dissertation.
We conclude this dissertation in Chapter 8 by summarizing and con- textualizing the contributions and discussing their implications for the eld of mobile human-computer interaction.
19
Chapter 2
Background and Related Work
In this chapter we survey previous work that is related to this thesis. It provides an overview of the state of the art in sensor-based interaction, in order for the reader to better classify the contributions presented in this work. The ordering of the related work sections in this chapter roughly follows the structure of the rest of the thesis.2.1 Continuous and Model-Driven Interfaces
Speed-dependent automatic zooming (SDAZ) is the navigation tech- nique that is the foundation of the work presented in Chapter 3 Con- tinuous Interaction and State-Space Systems.
SDAZ was discussed by (Igarashi, 2000) in 2000 as a technique for navigating large documents. They conducted a preliminary user study comprising a 1D document scrolling task and a 2D map navigation task and compared SDAZ to traditional pan-and-zoom navigation. Whereas SDAZ was clearly preferred by the subjects in the 1D scrolling task, the preference for SDAZ in the map navigation was only slightly higher than traditional navigation. Moreover, the authors did not observe an improvement in task completion time using SDAZ in either the 1D or 2D tasks. The authors used a reciprocal function with the scrolling speed as denominator to automatically control the magnication level. (Cockburn and Savage, 2003) conducted a similar study with a larger number of participants. They, too, compared traditional with SDAZ navigation (with a linear mapping from scroll speed to zoom level) for 1D document scrolling and 2D map scrolling. Interestingly, their re- sults are signicantly in favor of SDAZ, both in terms of task comple- tion times and NASA TLX (Hart and Staveland, 1988a) workload as- sessments. In contrast to the work presented in this thesis, Cockburn
20 2 Background and Related Work
et al. used a 2D map display with xed boundaries, which is unlike modernslippy maps, which we used for our studies. Slippy maps are used by most online map providers such as Google Maps, for instance. Slippy maps allow for innite scrolling in latitude and longitude, with a seamless rollover at the map content boundaries.
In contrast to previous studies on SDAZ-based interfaces for map nav- igation, the map interface we used contains real map material that cov- ers a substantial geographic area in a wide range of zoom levels. To our knowledge, such a realistic testbed has not been used for previous studies on mobile devices.
(Wallace et al., 2004) evaluated SDAZ scrolling speeds for a 1D text scrolling interface. Comfortable SDAZ scroll speeds are dependent on the assigned task (i.e., reading generally allows higher movement rates than looking at abstract data), visual perception, and the size and reso- lution of the device’s display. For our research, we could not apply the results of that paper directly due to the different target domain (mo- bile map navigation as opposed to text document scrolling) and because their study was conducted using a 19 inch monitor connected to a desk- top PC. We did however include some high-level practical advice, such as preferring an early zoom-out, in the interface for our study. Cock- burn et al. conducted a similar study, comparing SDAZ and several variants of SDAZ, (van Wijk and Nuij, 2004), with traditional scroll bars in a 1D document scrolling task.
A paper with signicant relevance to our work isOrthoZoom Scroller(Ap- pert and Fekete, 2006), a 1D scrolling technique that uses one mouse axis to input the panning speed and the orthogonal axis to control the content’s zoom (a similar technique,GestureZoom, had been previously presented by Patel et al. (Patel et al., 2004a)). Appert’s results have shown that OrthoZoom can be twice as fast as SDAZ. The zoom-level slider in our SAZ prototype (Section 3.1.2.6) interface has a similar function as the orthogonal axis in OrthoZoom and can also be consid- ered an orthogonal input dimension. In contrast to OrthoZoom, the slider in our implementation does not totally override the automatic zooming behavior but only sets the base zoom level for SDAZ to oper- ate on.
A study comparing pen-based rate control for map scrolling using SDAZ, pen-pressure-based zooming and tilt-based zooming was con- ducted by (Büring et al., 2008). The authors conducted their study on a tablet PC with the content scaled to the screen resolutions of mobile devices. In contrast to our study, they did not use tilt for rate-control and their map content covered a much smaller geographic area, which is likely to have reduced the amount of zoom levels needed. In their