Beyond Flat and Solid Interactive Surfaces

Despite their differences in size and usage contexts, all direct-touch interactive surfaces men- tioned above (e.g. touchscreens, tabletops, interactive walls) have one thing in common: they provide flat, non-deformable, rigid surfaces to the interacting user’s fingertip or hand. In this section, I will present approaches that break open this limitation. This enumeration of inter- face examples is organized in no chronological order and is targeted to illustrate the trend to- wards the vision of ’organic user interfaces’ [Vertegaal and Poupyrev, 2008] and ’radical atoms’ [Ishii et al., 2012]. I propose a classification in figure 2.8.

Non-Flat

The growing integration of interactive surfaces such as tabletops and interactive walls in office scenarios or public environments has brought up concepts for the seamless transfer of informa- tion from on device and surface to the other (e.g. [Rekimoto and Saitoh, 1999]). In addition to the transfer of digital information across different panes, researchers have proposed to physically merge interactive surfaces of different orientations. TheStarfireprototype presented as a concept video by Tognazzini in 1994 [Tognazzini, 1994] illustrates the vision of a non-flat interactive sur- face of the year 2004. The systems offers merged horizontal and vertical planes. Another non-flat,

2.2 Types of Interactive Surfaces 27

ORGANIC

flat non-flat non-solid adaptable deformable transforming

interactive surface

dynamic

Figure 2.8: My vision of the ongoing evolution of interactive surfaces towards deformable and actively transforming surfaces and interfaces. Non-flat interactive surfaces can unify properties of vertical and horizontal touch surfaces.Non-solidinteractive screens can allow for increased tactile expressiveness and input mechanisms such as force sensing. Interactive surfaces can be madeadaptableto the intended interaction and to the manipulated digital information by applying tangible user interfaces. Deformabletouch surfaces present their ductility as a means of input. Finally, transforming interactive surfaces flexibly develop input mechanism and manipulable materializations of underlying digital information. The interactive systems in this chapter are indexed according to this classification.

rigid interactive surface is the Sphere prototype presented by Benko et al. [Benko et al., 2008] (see figure 2.9). They present technical and conceptual solutions for non-flat multitouch surfaces. TheCurveprototype by Wimmer et al. is an digital desk which "combines vertical and horizontal working areas using a continous curved connection" [sic] [Wimmer et al., 2010] (see figure 2.9). They motivate their work with ergonomic considerations, flexible data transfer from the vertical to the horizontal panel and a suitability for different tasks. Roudaut et al. [Roudaut et al., 2011] evaluate the curvature of non-flat touch surfaces and show the influence of surface curvature and slope on input error rates.

Non-Solid

In the next step, non-flat interactive surfaces can step away from the surface’s rigid- ness and utilize passive material properties. Examples include properties such as flexi- bility using transparent rubber interfaces [Sato et al., 2009], softness with furry interfaces [Nakajima et al., 2011] (see figure 2.9) or flexibility using interactive surfaces made from dra- pable cloth [Lepinski and Vertegaal, 2011]. The object properties are used to enrich the input utilizing distance sensing or measurement of deformation.

Another approach to extend the interactive surface is to use both touch display and the adjacent space as a ’continuous interaction space’ [Marquardt et al., 2011]. Marquardt et al. describe sev- eral scenarios in which touch, gestures, and tangibles are used as input in this continuum and are moved between surface and environment. Also Hilliges discusses above-the-surface interac-

28 2 Interactive Surfaces

Figure 2.9: Non-flat and non-solid interactive surfaces: a: the Sphere prototype [Benko et al., 2008], b: the interactive desktopCurve[Wimmer et al., 2010], c: theFuSA2, a furry multitouch display made of plastic fiber optics [Nakajima et al., 2011]

tion and presents tracking techniques to "unlock the space above digital tables for interaction" [Hilliges, 2009].

Adaptable

In the next step, the form or composition of the interactive surface can be actively altered by the user to fit his needs. An example is a dynamically placeable semi-transparent interactive display which extends the display and shows parts of a 3D model [Chia-Hsun et al., 2003]. Of course, the concept of tangible user interfaces helps to extend digital information into the physical world and to customize the interactive surface for specified tasks. The TUI can be seen as an adaptable physical extension to the interactive surface. This extension can be general-purpose or bound to a certain task [Ullmer and Ishii, 2000] (see section 1.1).

Deformable

Standard TUIs such asUrp[Underkoffler and Ishii, 1999] which embody digital information are physical artifacts that lack the ability to change their shape. Ishii elaborates on the drawbacks: "Users must use a predefined finite set of fixed-form objects (building models in this case) and change only the spatial relationship among them, not the form of individual objects. All tangi- ble objects inUrpmust be predefined (physically and digitally) and are unable to change their forms on the fly" [Ishii, 2008]. Novel TUIs composed of continuous tangible materials such as clay or sand are used to embody and control digital data in non-solid but deformable repre- sentation. The projectIlluminating Clay[Piper et al., 2002] incoporates laser distance scanners and projectors to allow for rapid form sculpting (see figure 2.10). The system is used to digitally enhance a deformable clay landscape model with properties such as simulated water flow (see fig- ure 2.10). Other non-flat interactive surfaces which are deformable by the user consist of paper [Makino and Kakehi, 2011] or even the human body surface [Harrison et al., 2011] (see figure 2.10). Dynamic projection (e.g. [Sukaviriya et al., 2004]) and enhanced sensing devices (e.g. [Sato et al., 2012]) allow for the usage of arbitrary physical surfaces or objects as touch inter- faces. The notion of deforming interactive surfaces can also be used as input technique. Gummi

2.2 Types of Interactive Surfaces 29

Figure 2.10: Deformable touch surfaces: a: Illuminatic Clay[Piper et al., 2002], b: phys- ical paper used as deformable display [Makino and Kakehi, 2011], c: bendable computer prototypeGummi[Schwesig et al., 2004]

by Schwesig et al. [Schwesig et al., 2004] is a deformable mobile device prototype incorporating a flexible OLED display, bend sensors and circuitry. Bending the device as a whole is translated into zooming and scrolling interactions on the device (see figure 2.10). Thus, theGummi pro- totype closely follows the notion of ’organic user interfaces’. A very recent development is the

PaperTabsystem17, a lightweight tablet-sized plastic display, which recognizes touches, bending and folding as input mechanisms.

Transforming

The last class of interactive surfaces presented here can actively transform themselves to embody digital information or to match a certain task. Actuated workbenches actively move objects in two dimensions. Shape displays create 3D physical shapes directly or alter their tactile surface characteristic. Tactile tangibles, tactile displays and shape displays heavily rely on the creation of artificial tactile cues as an additional channel of information. Therefore, they will be discussed in detail in chapter 3.

In 2012, Ishii proposes the term ’Radical Atoms’ to denote reconfigurable particles of future materials. In this vision, all digital information will have physical representation and is directly manipulable. The envisioned development from GUIs over TUIs to ’Radical Atoms’ is depicted in figure 2.11. Ishii repeats the notion of ’tangible bits’, the physical embodiment of digital infor- mation for manipulation. However, he states that unlike pixels on the screen, physical tangibles do not change their form, color or position. The vision of ’Radical Atoms’ describes hypothetical dynamic materials, which

"Transformits shape to reflect underlying computational state and user input;

Conformto constraints imposed by the environment and user input; and

Informusers of its transformational capabilities (dynamic affordances)" [Ishii et al., 2012]. ’Radical Atoms’ are not restricted to flat, interactive surfaces but can be a form that is sensi- tive to touch or gestural input or that forms a tool-like device. The concept is closely related to

30 2 Interactive Surfaces

theClaytronicsproject [Aksak et al., 2005] which develops intercommunicating modular robots that can dynamically form 3-dimensional displays of electronic information. Ishii envisions the future interface as materials we can interact with. "We may call these human-material interac- tions (HMIs) or material user interfaces (MUIs), in which any object - no matter how complex, dynamic, or flexible its structure - can display, embody, and respond to digital information" [Ishii et al., 2012]. This vision of future interfaces heavily incorporates direct touch and manipu- lation and thus can be seen as a possible future for interactive surfaces.

Figure 2.11: From GUIs over TUIs to ’Radical Atoms’: Hiroshi Ishii’s vision of dynamic physical materials [Ishii et al., 2012]

I share this vision of more versatile interfaces which tightly integrate the concept of physicality and embodiment and allow for more diverse interactions and manipulations. The provision of programmed tactile feedback is an essential component of this development. The next section summarizes current technical and conceptual problems of interactive surfaces. Tactile feedback can help to address these challenges.