Flexibility

One fundamental problem is that the behavior of on-screen elements appears to be physical or realistic only at first sight – and this helps in reducing the learning threshold – but is pre-

46 3. Gesture based Interaction on Tabletops

programmed or scripted after all. The underlying problem here is that the user interface is subject to a limited set of pre-defined reactions to user input. Therefore, the designers and programmers had to anticipate each users needs at any given moment in any given situation. When observ- ing the frequency and creativity with which participants made use of the affordances of simple tools such as pen and paper in the real world condition it becomes quite obvious that not all of these could have been anticipated in advance let alone the difficulties in sensing, recognizing and differentiating subtle changes in user interaction. In hindsight we would argue that the pseudo- physicality in theBrainstormsystem was successful as measurement to lower the initial learning threshold but can be a limiting aspect in terms offlexibility.

The following aspects illustrate as to why and how this lack of flexibility can be a problem. While we have discussed benefits for learnability of our real-world inspired gesture-based in- terface this design choice raised false, sometimes problematic, users’ expectations. This led to frustrations when the system did not respond as they expected based on their real-world knowl- edge. In the simplest cases this was a matter of gesture recognition; in the real world we have a variety of strategies to push objects around. We can touch them from the top and drag them or we can use one or more fingers to push objects from the side. Brainstormonly implemented the former and we frequently observed users trying to perform the latter – often repeating their attempts under growing frustration and comments that this "should work like this if it was the real thing". Clearly here a mismatch between implementation and mental model can be observed.

(a) (b)

Figure 3.7: Problems with explicit mode switching in Brainstorm: (a) Awkward posture for handwriting. Only one hand is used to write on paper. (b) Artifacts caused by the user trying to move the post-it while in writing mode.

In other cases the need to recognize certain user intentions and the limitations in the sensing technology caused similar problems. For example, input inBrainstormhappens through a single point of contact (per user) so that for many functionalities we had to design abstract or iconic gestures to switch modes. The most prominent example is how users have to switch between writing on post-its and moving them. A double tap within a designated area enlarges post-is and makes them writable but fixes them to their location, a second double tap shrinks them and makes them mobile again. We frequently observed how this mode switch caused problems (even after

3.5 Observations and Implications 47

extended use). Often users would try to move objects when in writing mode, causing unwanted inking across the written text or users tried to quickly scribble something onto post-its without enlarging them first (see Figure 3.7). In the latter case users often tried to hold onto the post-it with the second hand to hold it in place for writing (before they remembered that they had to double-tap post-its to switch modes).

3.5.3 Summary

In summary we would argue that exploiting physical affordances in the virtual interface was successful to a certain degree, especially as a learning aid. However, some of our design choices turned out to be problematic. These can be categorized in two ways. First, over-simplifying New- tonian physics caused user frustration especially when they could not apply real world strategies to achieve seemingly simple things such as moving objects but had to perform a specific gesture in a particular, pre-defined way. Second, the resemblance of the interface to physical objects seemed to encourage users to interact with the system in more analogue, richer ways than just through a single point of contact. We observed bi-manual interactions and attempts to use mul- tiple fingers at once to manipulate one object or even whole hands to move several objects at once.

From our exploration into gesture-based interaction we can distill the following key aspects to inform the next steps in our research endeavor:

• _{Real world resemblance and metaphoric gestures can lower the learning threshold. This}

interaction style could be appropriate for applications with limited feature sets and casual application domains so that problems with long term memorization are not as important.

• _{The need for gesture recognition (and therefore anticipation of user intentions) can be}

a limiting factor in terms of interaction flexibility. In the worst case this limitation can frustrate users and render a design in-successful.

• _{Using a pen or stylus in combination with pen-trace gestures does not mimic the rich ways}

of interaction we enjoy when manipulating real world objects sufficiently.

• _{Limiting input to a single point of contact can be problematic in some situations - especially}

if graphical elements resemble real world objects but can only be manipulated through a pointer based interaction model.

These observations are the main motivation for our next exploration into tangible objects in combination with interactive surfaces to further unlock our manual dexterity.

Chapter 4

Tangible and Hybrid Interaction on

Tabletops

Tangible user interfaces (TUIs), inspired by the seminal work of Fitzmaurice et al. [FIB95] ex- pand the interaction vocabulary by exploiting fine grained motor skills that humans possess when manipulating tangible objects. The main benefits claimed in this area of research are intuitive- ness [IU97], motor memory [KHT06], learnability [RMB+98] and the possibilities of conveying the rich meanings in social settings [HB06]. Some TUI examples are literal instantiations of metaphors [UI97, UI99b] where the physical and the digital are tightly coupled. Other varia- tions allow for more generic mixed physical and graphical interaction [RUO01]. Often uses of the tangible paradigm are motivated by the goal to support co-located collaboration, for example TViews [MRD06], and Urp.

Recent hardware advances have made it feasible to sense and identify tangible objects on tabletop displays. Wilson [Wil05] demonstrates a vision-based system capable of tracking phys- ical objects through visual barcodes, hands and sheets of paper using IR-illumination and an off-axis camera equipped with an IR cutoff filter. A similar technique is used in the reacTable [JGAK07] to track objects that serve as input to a virtual musical instrument. This hybrid ap- proach to interface design appears as particularly promising because of two reasons: First, direct- touch interaction alone is an intuitive and easy to learn interaction style. However, the bandwidth of input and the interaction vocabulary are limited in comparison to the flexibility we enjoy when interacting with the real world. Second, in the light of our analysis of a purely gesture-based approach (cf. 3) and recalling the user frustration caused by the limited degree of physicality

in the purely virtual interface of the Brainstorm system using physical objects as handles for interaction appears as an interesting avenue for exploration.

In this Chapter we describe our exploration of hybrid interaction (i.e., tangible interaction combined with direct-touch interaction) usingPhotohelix[HBB07], a system designed for shar- ing personal photo collections on a digital table, as vehicle. We report findings from several user studies. At the end of this Chapter we furthermore discuss the suitability of this approach as a general interaction model for tabletop computing.

50 4. Tangible and Hybrid Interaction on Tabletops

4.1

Motivation

Interactive surfaces and digital tabletops in particular, offer a compelling platform for shared dis- play collaboration, allowing multiple users to interact simultaneously with a shared information landscape. These platforms provide the same (social) functions as traditional tables allowing multiple people to sit around the table and share artifacts on the surface of the table only with the difference that the artifacts can be digital. An often heard explanation is that digital tabletops afford mutual eye contact and body language as well as other properties important for verbal and non-verbal communication such as the possibility to interact with and exchange data artifacts through direct-touch interaction.

One of the main advantages of interactive surfaces is the flexibility of the interface; because it is purely virtual the interface can be dynamically reconfigured to serve different purposes be it for work or play. However, touch sensitive surfaces do not offer the same tactile feedback which tra- ditional (non touchscreen) interfaces provide through physical buttons, knobs and switches. This feedback is important for motor learning and the automation of repetitive tasks such as touch typ- ing on a QWERTY-keyboard (text entry is a task still notoriously difficult on direct-touch inter- faces cf. [HHCC07]). Using physical controls in combination with interactive surfaces promises to combine the best of two worlds – dynamically reconfigurable graphical output coupled with the possibility to directly interact with digital content and at the same time physical handles for interaction. These promise to unlock the operation of an interface – or key elements thereof – without visual attention in a multitasking situation such as sharing of digital photos.

An understanding of the operation of user interface (UI) elements without looking at them (or only briefly) from an embodied cognition perspective [Cla00] would though suggest that the use of 3D elements at the interface offers a form of tactile feedback unavailable in a direct touch en- abled UI. This tactile feedback should theoretically redistribute cognitive effort into other sensory modalities. So whereas with the direct-touch UI the cognitive effort is largely expended through visual attention, with the tangible interface some of the effort of control (i.e. feedback loops gov- erning appropriate interface manipulation and understanding that you are still within the bounds of reasonable control movement) becomes tied to tactile interaction. We call thiseyes-freema- nipulation. To give a simple analogy we could consider driving a car. This is a visually intensive activity, visually monitoring the road and traffic is mandatory. If for example, the controls for driving required additional visual monitoring such as in a car that was completely drive-by-wire and the control mechanisms were based entirely on a direct-touch UI system, then the car would become too dangerous to drive. Consequently cars have maintained physical analogue controls which provide tactile feedback of their manipulations (e.g. steering wheels and pedals) so that the visual resources are kept free for relevant activity.

A second argument for the usage of physical handles is the expressiveness and flexibility provided by real 3D objects as input devices. Designing tangible interfaces promises to exploit fine-grained motor skills and exploiting 3D shape and mechanical constraints to implicitly convey the correct usage of an artifact. For example, a well designed door handle communicates without further instruction how to operate it. In addition to their main purpose physical objects can be repurposed to serve other uses. For example, imagine the plethora of applications a simple screw