touch screens, and can be accessed on Android devices (Google Inc., 2012a)W and (unofficially) on iOS devices (Stackoverow user Ken-
nyTM, 2010)W. Of course, these approximations are not as pressure
sensors such as force-sensing resistors (FSRs).
2.4.1
Mapping of Pressure to Input Values
Previous work has discussed different kinds of discretization and trans- fer functions to process raw sensor input in order to achieve better user performance. (Cechanowicz et al., 2007a) explored different dis- cretization functions for a pressure-sensitive mouse and found that a quadratic mapping centered at the lower range works best. In (Ramos and Balakrishnan, 2005) a parabolic-sigmoid transfer function was used. (Shi et al., 2008) found that a sh-eye discretization function was superior to other mappings.
2.4.2
Feedback for Pressure Input
Most pressure-based input techniques rely on continuous visual feed- back. Tactile and multimodal feedback play an important role in pen-based interfaces (Lee et al., 2004; Liao et al., 2006). (Rekimoto and Schwesig, 2006) implemented a three pressure-level button (“not pressed”, “pressed lightly” and “pressed hard”). The button provided tactile feedback upon corssing the boundaries of these levels. The ef- fectiveness of tactile feedback in mobile devices has been explored in (Brewster et al., 2007b; Hoggan et al., 2008; Luk et al., 2006). EarPod (Zhao et al., 2007) investigated auditory feedback for eyes-free inter- actions on mobile devices.
2.5 Gestural Input in Mobile Interaction
In Chapter 5 we present our work on mobile interaction using motion gestures. Since previous work on 2D stroke recognition is highly rel- evant, we will discuss it rst, followed by work on motion gestures, which are usually captured in 3D from tri-axis accelerometers and gy- roscopes.
26 2 Background and Related Work
2.5.1
2D Pen and Touch Gestures
The $1 Gesture Recognizer (Wobbrock and Wilson, 2007) is a recog- nizer for pen or touch strokes on touch-sensitive screens. The em- phasis of the $1 Gesture Recognizer lies on simplicity and ease-of- implementation. It does not require any external toolkits and recog- nizes input strokes robustly. The algorithm is so simple that its pseu- docode completely contained in the paper. Multistroke recognition was added the $1 Gesture Recognizer by (Anthony and Wobbrock, 2010), and adds several useful optimizations to the basic concept. In Section 5.2, we discuss the $3 Gesture Recognizer, an algorithm that builds on the concepts of the $1 Gesture Recognizer to allow the recognition of 3D motion gestures.
Protractor (Li, 2010b) is a gesture recognition algorithm for touch screens. It is a template-based recognizer that resamples the gesture trace to get a vector with a xed dimension, which is then translated to the origin and normalized. For each comparison with a template a rota- tion is performed that optimally aligns the input and template gestures. This is done in an efficient way using a closed-form analytic approach, in contrast to the$1 Gesture Recognizer. Protractor3Dextends this con- cept to use a closed-form solution for optimal rotational alignment for 3D motion gestures (see Section 5.3).
Giving continuous feedback and also feedforward during gesture stroke input can be important, rstly because novice users can more effectively learn the gesture stroke vocabulary, and secondly because they get the opportunity to correct their inputs. OctoPocus (Bau, 2008) is a gestu- ral input system for menu selection that provides both feedback and feedforward, and the authors show that OctoPocus performs favorably compared to a conventional hierarchical menu.
2.5.1.1 Use of Motion Gestures in Mobile User Interfaces
RexPlorer(Ballagas et al., 2007) is a location-aware mobile game that allows tourists to explore the city of Regensburg, Germany. The in- teraction is based on movement gestures: The user can cast different types of “spells‚ to solve a riddle and learn more about the city”. Rex- Plorer uses very simple u-shaped gestures and an ad-hoc recognition scheme (Kratz, 2007b). The motivation for using gestures with simple shapes was to guarantee the gesture recognition when using noisy data for input. Motion information for gesture entry was obtained from 2D motion estimation of images from the mobile device’s built-in camera, thus the system can only recognize motions based on 2D shapes.
2.5 Gestural Input in Mobile Interaction 27
(Kela et al., 2006) developed an accelerometer-based input mechanism for interaction in a smart workplace for designers and engineers.
DoubleFlip (Ruiz and Li, 2011) addresses the problem of false positive gestures by employing a simple delimiter gesture (the “double ip” gesture) for entering gesture input mode. The criteria for choosing the “double ip” gesture were that (1) the gesture had to be easy to perform and (2) the gesture had to be sufficiently distinct from every- day movement. To conrm the latter the authors investigated the false positive rate for a large corpus of everyday movement data.
(Hinckley and Song, 2011) explored how to combine touch input with motion gestures. This kind of multi-modal interaction opens up a de- sign space that allows very expressive kinds of mobile interaction. In particular, the accelerometer can be used to gain additional information about touch events, i.e., “soft touch” or “hard touch”.
2.5.1.2 Vocabularies for Interaction via Motion Gestures
Li et al. (Ruiz et al., 2011) applied the approach of Wobbrock et al. (Wobbrock et al., 2009) to mobile gestures. The idea is to confront users with a desired task goal (such as deleting items) and let users choose which gesture they nd the best match for the goal. (Kuehnel et al., 2011) followed the same approach for gestural interaction in smart-home environments.
2.5.1.3 Gesture-Based Authentication
In Section 5.4 we present work on gesture-based authentication. There have been several previous publications on this topic.
(Patel et al., 2004b) created a gesture-based authentication scheme to authenticate mobile devices on public terminals via shaking. When the user selects a device to authenticate with, the device generates a gesture input prompt. If the user performs the gesture as requested, the device is authenticated an the user can interact with the public terminal via the authenticated mobile device.
(Farella et al., 2006) studied gesture-based authentication using an accelerometer-equipped device. The authors particularly focused on di- mensionality reduction, comparing the results of PCA¹ and LLE² with
¹principal component analysis (Jolliffe, 2002). ²locally linear embedding (Saul and Roweis, 2000).
28 2 Background and Related Work
ak-nearest neighbor classier. In their study, PCA achieved better re- sults than LLE. The overall results of the study lead the authors to the conclusion that gesture-based authentication is feasible given a small user population size. However, the authors did not evaluate the resis- tance of their system to visual attacks.
(Mayrhofer and Gellersen, 2007) developed a scheme for device-to- device authentication based on synchronous shaking. Acceleration data from the devices’ accelerometers was used as a basis to generate and authenticate cryptographic keys. This general idea has also been de- veloped further and commercialized in form of the Bump application (Wikipedia, 2012)W, which allows the exchange of contact information
between to smart phones by bumping them together.
Liu et al. evaluated gesture-based authentication, based on theuWave system by the same authors (Liu et al., 2009b,a). Their results show that movement gestures can be remembered at least as well as stan- dard pass phrases. Furthermore, they tested the robustness of their al- gorithm towards visual disclosure of the gesture entry and found that visual disclosure signicantly raises the false positive rate.
2.5.2
Recognition Systems for Mobile Motion Gestures
uWave(Liu et al., 2009b) is a motion gesture recognizer based on Dy- namic Time Warping. The accuracy obtained by their system reached 98.6%. The paper suggests a template adaptation system, where a weak matching templates are periodically replaced by the current gesture in- put, in order to counter subtle changes in the user’s gesture entry tech- nique over time.
(Schlömer et al., 2008) presented an HMM-based classier for gestures input via a WiiMote. The accuracy achieved is, however, does not ap- pear to be greater than that of simpler, template-based approaches.
(Hoffman and Varcholik, 2010) compared a linear as well as an Ad- aBoost classier for motion gestures input via aWii Remote³. They ana- lyzed the effects of adding rotational features to the for recognition by using data from the gyroscopes on theWiiMote MotionPlus⁴. Although our work in Section 5.5 partially replicates the study of the effects of gyroscope data, the sensor setup of the motion plus is different from that typical of mobile phones, as it has one two-axis gyroscope and one ³The WiiMote is a game controller manufactured by Nintendo that incorporates a 3-axis accelerometer (Lee, 2008).
⁴The MotionPlus is an attachment for the WiiMote that adds gyroscope to the Wi- iMote. (Wii Brew, 2012)Wprovides extensive information on sensor information pro-
2.5 Gestural Input in Mobile Interaction 29
single axis gyroscope. Mobile phones usually contain a single three- axis gyroscope. Moreover, we also focus on methods for combining accelerometer data with gyroscope data for algorithms that do not sup- port feature dimensionalities larger than 3.
2.5.3
Development Tools for Gesture-Based Interfaces
In Chapter 6, we presentMayhem. Although Mayhem is not explicitly designed as a development tool for gesture-based interaction, it can be used to rapidly connect the output of gesture recognizers to complex functionality, eliminating the need for additional software development to attach functionality to gesture recognizers. This can be very useful, e.g., when prototyping gesture-based user interfaces. In the following, we discuss two related systems that, in contrast toMayhem, focus more on developing the actual gesture vocabularies and gestures used for input.
SATIN (Hong and Landay, 2007) was a development toolkit for pen- based user interfaces. Built on top of Java Swing (Fowler, 1998)W,
SATIN tried to help developers of such interfaces by providing a scene graph, rotation and scaling support for virtual objects and also stroke recognizers and interpreters for pen-based strokes.
MAGIC (Ashbrook, 2010a) is a development tool for the design of gesture-based interfaces. It specically addresses developers that have no knowledge of pattern recognition or machine learning. Additionally, Ashbrook and Starner recorded an “Everyday Gesture Library” that al- lows developers to test how well their gesture designs can be disam- biguated from everyday movements.
31
Chapter 3
Continuous
Interaction
and
State-Space Systems
“Any sufficiently advanced technology is indistinguishable from magic.” —Arthur C.Clarke A number of mobile interaction tasks require the user to repeatedly per- form the same input gesture in order for him to accomplish the tasks. Examples of this are mostly found when the user has to navigate an ex- tensive set of data, i.e. navigating a lengthy list of contacts or scrolling through a long web page. A further example of such tasks is map nav- igation. Here, the space the user is navigating is extremely expansive, and scrolling takes place in two dimensions and at a much larger scale than scrolling in lists or multi-page documents.
Concretely, multi-touch user interfaces exhibit a weakness when users are tasked with navigating large data sets such as maps, forcing the users to use a high number of “ick” inputs to scroll as well as “pinch” inputs to zoom in or out of the map.
Our motivation for the UI concepts and user studies presented in continuous input and automated control of certain UI parameters may lead to lower effort and better usability this chapter is to use the sensing capabilities found on existing mo-
bile devices as well as dynamic UI models based on state-space sys- tems to reduce the number of input repetitions needed for mobile map navigation, in order to increase the usability. Our two main strategies to accomplish this goal are to use continuous input vs. dis- crete input and to allow the UI to assist in controlling the zoom level and scrolling speed, using a state-space Speed-Dependent Automatic Zooming (SDAZ) model. The motivation behind automating a number of important user interface parameters is that by reducing the amount
32 3 Continuous Interaction and State-Space Systems
of inputs the user needs to make, his effort is reduced, thus freeing some his mental capacity to allow him to better focus on the goal he wants to achieve by performing the navigation task.
In Section 3.1 we introduce Semi-Automatic Zooming (SAZ). SAZ uses tilt information obtained from the mobile device’s accelerometer in or- der to control zoom, scroll speed and scroll direction. SAZ builds upon SDAZ by adding the capability to override automatic zooming at any time through a slider that controls the base zoom level.
Motivated by the results obtained from our user evaluation of SAZ, we evalute if automatic zooming can be integrated into a standard multi- touch interface in a more subtle way. This approach aims to combine the advantages of these two techniques in a single map navigation in- terface. Section 3.2 presents a technical description of such a system and the results of preliminary we conducted to evaluate it.