Bimanual Spatial Manipulation

Guiard argued that skilled manual actions generally involve both hands in an asym- metric division of labor. The non-dominant hand defines a spatial reference frame for more fine-grained actions of the dominant one [115] (see also Section 4.3). When slicing bread or vegetables, for example, one hand operates the knife, the other one holds the piece to be cut. Also the bimanual operation of tools generally follows this asymmetric division of labor: if the tool is meant to be operated with a forward thrust along its elongation, e.g., a spade, a drilling machine, or a billiards cue, the non-dominant hand stabilizes its posture and orientation at a more distal position, while the dominant hand exerts the necessary force. If the tool is instead meant to be operated with lateral motion, e.g. a golf club, a hammer, or a rake, we prefer to hold it the other way around to enable more controlled and forceful lateral movement through the dominant hand (Figure 5.1).

Figure 5.1: The bimanual operation of mundane tools is often asymmetric. For-

ward thrust, e.g., when using a spade, is generally performed with the domi- nant hand at the far end of the tool handle. The non-dominant hand supports with guidance at a more distal working point. The placement of hands is usu- ally the other way around if lateral tool movements must be controlled, e.g., when using a pick axe. A pushcart is one of the few devices that are operated with both hands in equal roles, to facilitate balance and apply more force.

Consequently, most bimanual computer interfaces that have been proposed also build on Guiard’s model of a kinematic chain. In the realm of computer inter- faces, input from the non-dominant hand has been suggested to control the con-

text in which selection and manipulation input of the dominant one is meant to be applied. Examples include the view at a digital document or a virtual environ- ment (e.g. [135, 375]), the handling of a 3D object for closer examination and edit- ing (e.g. [137, 138, 314]), the definition of motion constraints like a pivot point for rotation and scaling (e.g. [62, 67, 173, 314, 375]), or the provision of additional func- tionalities via menus or magic lenses (e.g. [38, 314]).

Unfortunately, only few of these developments are available in commercial products. The research prototypes used different input sensors or tracking markers for both hands, e.g. touch and pen [44, 143] which are often missing on commodity devices. More recently, devices with touch and pen input have become available, but unfor- tunately, the combination of both inputs is not yet well supported. They are often interpreted equally and mutually exclusive.

Symmetric bimanual interaction techniques have also been proposed. Most of these were developed to increase the number of simultaneously available degrees of freedom. Established single-pointer input is limited to the selection of coor- dinates and translation input. With more than one access point manipulated ob- jects can also be rotated, scaled, and even deformed in a single coordinated action (e.g. [67, 100, 135, 200, 201, 203, 208, 236]). Note that also in case of such symmetric in- teraction of both hands, a tendency towards asymmetric operation can be observed. If it comes to fine tuning the manipulated parameters, the hands alternate between asymmetric roles of holding a reference position and motion input.

The combination of rotation, scaling, and translation input (RST) with two contact points has become a basic building block of multitouch interfaces for 2D and 3D ge- ometric manipulations (e.g. [181, 280]). On mobile touch devices, these gestures are often performed with two fingers of one hand. Whether two hands are employed or two fingers of one hand primarily depends on the size of the screen and the manip- ulated content. Asymmetric touch input, however, is less established. The following subsection discusses research efforts to facilitate asymmetric bimanual interaction with multitouch devices.

Asymmetric Multitouch Input

Multitouch input sensors implicitly provide a consistent spatial reference frame for motion input from multiple fingers. If the sensors are applied as a display over- lay, they even blend motor and display space to a coherent whole. Therefore, these devices are well suited for various bimanual interaction techniques. Surprisingly though, existing multitouch systems make only limited use of these capabilities. Only the pinch-zoom gesture has become an integral part of commercial multitouch inter- faces. It is even more surprising that asymmetric interaction is not well supported, as it is more common in real-world settings [115].

Multimodal Interaction 65 One reason for this situation is certainly the context of use. Multitouch sensors are most established in mobile computing, where one hand is commonly required for holding the device. Thereby it already provides a spatial reference frame for input from the other hand. Corresponding to the often-cited example of writing on a sheet of paper, the non-dominant hand holds the medium on which the dominant hand is acting. With digital media, however, the held device is only a physical frame for dynamic digital content. The dominant hand navigates through the displayed infor- mation instead of editing it as in the case of handwriting.

For asymmetric bi-manual interaction with the digital content, the non-dominant hand must also provide a reference frame in the virtual interaction space. Several research prototypes have shown that this can be feasible despite the limited motor capabilities while holding a mobile device [95, 133, 224, 352]. Recent developments also showed how device motion and touch input from the same hand can be com- bined for more efficient navigation input. Integrated inertial sensors complement touch screen input with input options related to the selected on-screen target. For example, Hinckley et al. show how tilting the device can perform a zoom into a map region selected by touch [140].

Asymmetric bimanual touch input requires the discrimination of fingers and hands as a prerequisite for the assignment of different roles. This is also true for stationary devices, where both hands are fully available for cooperative actions. Touch sensors generally cannot discriminate different hands or fingers. Marquardt et al. experi- mented with finger-worn markers to explore the potential applications if each finger can be identified [220]. Holz and Baudisch realized a touchscreen as a large-scale fingerprint reader [146]. For more widely applicable hardware systems several dis- crimination heuristics on the basis of fingertip shape and hand posture have been suggested [13, 70, 354, 382].

Alternatively, applications can provide additional input modes for asymmetric bi- manual interaction via dedicated buttons and on-screen widgets (e.g. [62]). A more versatile solution is the combination of pen and direct touch input. The explicit dis- crimination of roles based on the used devices is robust and supports a wide range of powerful cooperative actions [44, 136, 143, 266, 374]. Research on bi-manual inter- action performed as part of this thesis showed that asymmetric bi-manual input can also be implicitly derived from the fundamental behavioral pattern that the reference- providing hand generally initiates the cooperative action (see Chapter 9).