Ishii’s tangible user interface model

Dix, Finlay, Abowd and Beale (2004) describe human-computer interaction as being concerned with the interaction that the user has with a computer in order to accomplish a task. According to Ghaoui (2005), and Sears and Jacko (2009), HCI is an interdisciplinary and multidisciplinary research domain that emerged from computing and includes contributions from (amongst others) computer science, psychology, cognitive science, ergonomics, sociology, engineering, education, graphic design, and

industrial engineering. Ishii (2008a, 2009) reminds us that human-computer interaction design principles have progressed from requiring the user to remember commands and typing these commands (as is the case of the so-called command user interface, also written as CUI), to pointing at a visible rendition of the command using a computer mouse and selecting the rendition by the press of a mouse button (as is the case of the so-called graphical user interface, also written as GUI).

The TUI provides an alternative to both the CUI and GUI by taking form as tangible objects that both represent digital data and operate as tools for direct manipulation of digital data. The TUI is in sharp contrast to the GUI in that the GUI multiplexes the interface mechanism (mouse) in both space and time whereas the TUI makes provision for a dedicated interface for the data being manipulated.

Another contrast is that the GUI is limited to intangible data representation whereas the TUI supports tangible representation.

Figure 3-3 depicts Ishii’s basic TUI model. This model associates tangible representations of data, digital data, or digital computations with physical objects. The model also makes provision for changes to the associated digital entities through manipulation of the physical objects (Ishii 2009).

Figure 3-3 Ishii’s basic tangible user interface model

(Ishii 2009)

He extended the model to include three feedback loops. The first is passive and exists between the user and the object that the user manipulates. This feedback is in immediate tactile form and exists independently of a computer. A second and active feedback loop is between the user and a computer program. The output of the computer program changes as the user manipulates the physical objects. Changes are fed back to the user in (for example) visual form. The third active feedback loop is between a physical object and data. This data may represent a digital model or it may be the result of computation and can be used to affect physical properties of the object (Ishii 2009).

What Ishii’s (2009) TUI model does not reflect is how the tangible representation is conceived and by whom. Indeed, the majority of the literature discussed in this study does not indicate the user’s

involvement in determining the tangible representation. It is therefore reasonable to assume that, in the majority of TUI-based systems, the user is not involved in the design of the tangible representation and has to adapt to a predetermined representation.

An alternative approach extensively investigated in this thesis considers the scenario where the user decides on the tangible representation. To this end, Ishii (2008b) describes an “organic” tangible representation that allows the user to shape material supplied by a system designer. Systems that incorporate organic tangible representation make provision for the TUI system designer to prescribe the basic properties of the tangible representation yet also allow the user to change the tangible representation. Piper, Ratti, and Ishii’s (2002) Illuminating Clay and Ishii, Ratti, Piper, Wang, Biderman, and Ben-Joseph’s (2004) SandScape are examples of TUI systems that include endlessly malleable materials to facilitate organic representations.

Chapter 6 describes my T-logo tangible programming system that includes certain aspects of Ishii’s organic tangible representation. When using the T-logo programming environment, the user is free to decide what materials to use when constructing the programming objects.

Fitzmaurice, Ishii and Buxton’s (1995) Graspable User Interfaces and Ishii and Ullmer’s (1997) Tangible Bits explored mechanisms that allow a person to directly “touch” data using graspable media. Ishii and Ullmer’s research categorised a user’s attention as being in either a “foreground” or

“background” state. They argued that when the user’s attention is in the foreground the user focusses on the task. Conversely, they argued that when a user’s attention is in the background then his attention is not centred on the task and the user remains aware of her immediate surroundings (the periphery). Ishii and Ullmer viewed the two states as being mutually exclusive. Their graspable media was designed to be used at the centre of a user’s attention and their ambient media was to be used at the periphery of the user’s attention. They also aimed to develop a type of human-computer interface that allows the user to “touch” data stored within the human-computer. They dubbed this type of human-computer interface a tangible user interface (Ishii & Ullmer 1997). Not only did they consider solid objects as potential tangible user interfaces but they also considered fluid-like mediums such as audio waves, visible light, flow of air, liquid, and gas that Ishii (Ishii 2009) called ambient media. These fluid-like mediums were recognised as possible TUIs and specifically for use in the background of a user’s attention. Ishii refers to objects that are both physical and graspable as tangible objects (Ishii 2009). My research considers a programming environment that requires focussed attention and the manipulation of solid objects.

A mapping between specific GUI elements and TUI elements emerged from research conducted during the 1990’s. Table 3-1 gives a selection of these mappings while Figure 3-4 illustrates them.

Table 3-1 A selection of mappings between Graphical User Interfaces and Tangible User Interfaces Graphical User

The GUI window is mapped to a tangible frame, called a lens. The lens can be positioned in physical space, with a display within the frame changing its projection under software control.

icon phicon The GUI icon is mapped to a tangible object that represents specific digital data and can be positioned in space.

menu tray The GUI menu is mapped to a physical tray that may contain one or more physical objects, with each object a selectable item.

handle phandle The GUI handle (used in changing the size of a GUI window) is mapped to a tangible object called a phandle.

widget instrument The GUI widget (that is often used in adjusting linear quantities) is mapped to a tangible slider mechanism called an instrument.

(Ishii & Ullmer 1997)

Figure 3-4 A selection of mappings between graphical user interfaces and tangible user interfaces

(Ishii & Ullmer 1997)

According to Ishii (2008b), researchers developed TUIs that incorporate tangible materials of which the shape is integral to the role of the TUI. An example of a TUI of which the shape is significant to its meaning is the tangible equivalent of the GUI widget (see Figure 3-4, bottom right). In addition to the research into fluid-like TUIs for use at the background of a user’s attention, second generation fluid-like TUIs were also developed for applications at the centre of the user’s attention. Examples of second generation TUIs include sand and clay. These fluid-like TUIs can be reshaped, with their digital representation changing at the same time. In addition to some TUIs that can be reshaped by the user, other TUIs can be reshaped by software through a process called actuation. Lumen (Poupyrev, Nashida, Maruyama, Rekimoto & Yamaji 2004) and Ohkubo, Ooide, and Nojima’s (2013)

“smart hairs” are examples.

Of particular interest to my own study is Ishii’s consideration of objects that can be grasped and are found in the home or office environments. Ishii suggested that such objects can be used at the centre of a user’s attention. He and Ullmer were particularly interested in exploiting the rich affordances that physical objects offer. Ishii also considered the potential of architectural surfaces serving as tangible user interfaces, dubbing these interactive surfaces. Such surfaces were to be used at the centre of a user’s attention. Examples of interactive surfaces include those found as part of a

building structure (an office wall is an example) and those that serve as furniture (the office desk, for example). Having now established that physical objects can represent and manipulate data, it is worthwhile to consider what meanings can be attributed to physical objects.