Abstracting Frameworks

Abstracting frameworks help researchers to better understand the properties of tangible interaction and propose models and taxonomies to frame the typical interactions that can be found in existing tangible interactive systems or that can be designed for future systems. Most frameworks in this category, according to the classification of Mazalek and Hoven [137], focus on the facets of physicality i.e., the

Figure 4. Classification of Tangible Interaction frameworks according to abstracting, designing and building types [137]

artifact design and the affordances provided by object forms, and interactions, i.e., the interplay between the user and the digital systems as well as the links between physical objects and actions and digital objects and effects. In this section, first, I present the most popular interaction models for tangible user interfaces. Then, I present the existing taxonomies for classifying objects in tangible interaction.

The first model that helped understanding the main principles of tangible interaction was presented by Ullmer and Ishii in 2000 [205]. They opposed the Model Control Representation physical/digital (MCRpd) of Tangible User Interfaces (TUIs) to the Model View Control (MVC) of Graphical User Interfaces (GUIs), showing that in TUIs digital information has a physical representation, which is not present in GUIs, and that the physical representation is also the control (Figure 5). Indeed, the users can benefit of the tangible artifacts to have an immediate understanding of the state of the underlying digital information and they can also directly manipulate this information by means of the tangible artifacts. The digital information has not only a physical tangible representation but also a digital intangible representation, e.g., video projection on a tabletop. Although intangible representation could not be grasped, it is generally directly coupled to the tangible artifacts and can be also manipulated through this latter.

Figure 5. MCRpd model for Tangible Interaction [205]

In an updated model [97], Ishii explained that the human-computer dialogue in TUIs happens through three possible loops: in the first loop, the users’ actuation has immediate feedback through the tangible representation of the information, i.e., the artifacts that they have just manipulated; in the second loop, the computer provide additional feedback by sensing the artifacts movements and by

computing and updating the intangible representation; in the third loop, the computer is able to communicate with the users by actuating the tangible representation of the information, i.e., by moving the physical artifacts (and updating the intangible representation accordingly) (Figure 6). Possibly, the physical artifacts can be as small as physical pixels: in radical atoms, the actuation is done at atomic level and the users can control the form of the matter through direct manipulation but also

through gestures that are not directly coupled with the matter deformation, with feedback actuated by the computer after the interpretation of the users’ gestures.

Shaer et al. [187] proposed a model based on token and constraints (TAC), where tokens are physical artifacts that represent a digital information or a computational function, while constraints are all the physical elements of the interface that limit the interaction space and provide the user with affordances about how to interact with tokens. Shaer and Jakob [186] further developed the TAC paradigm describing with a formal markup language (TUIML) the interactions of Tangible User Interfaces, in particular of those that allow the manipulation of data for professional purposes. They modeled the behavior of a Tangible User Interface through a two-tier model: the dialogue tier is a state-transition diagram that represents all the distinct states of the application, each state differing from the others in the interactions that the user can perform. The interaction tier details each interaction for a particular task or function and depicts the user elementary action in the physical world and the computer responses in the digital world.

A consistent part of tangible interaction frameworks and taxonomies focused on the classification of the object types. Holmquist et al. [90] proposed a first classification of objects according to their role on the system and their physical resemblance to the digital objects. They classified objects into containers, which are carriers for digital information, tokens, which physically represent a digital object thanks to some form of resemblance, and tools, which allow manipulating digital information often through a computational function. Ullmer and Ishii [205] further extended this classification of objects by introducing the idea of dynamic binding, a property that allows the users to dynamically associate digital information to a general purpose container, and a semantic

Figure 6. Interaction loops in tangible interaction [97]

classification of objects into iconic and symbolic, whether the link with the digital object is done through physical resemblance or by an assigned symbol, respectively. Moreover, they classified systems according to the role that objects have in the system, i.e., the relationships between physical and digital: spatial, constructive, relational and associative systems. Koleva et al. [119], rather than individuating specific classes of object, classify tangible artifacts according to the degree of coherence between the physical and digital objects (Figure 7). For example, generic purpose tools that can be assigned to different functions have a low degree of coherence, while artifacts that are coupled directly to digital objects and that allow manipulating them physically have a higher degree of coherence.

Koleva et al. [119] individuated several factors that can affect the degree of coherence, in particular, according to the properties of the relationship between physical and digital artifacts, e.g., whether the relationship is permanent or not (cf. the dynamic binding of Ullmer and Ishii [205]), whether the action performed on the physical objects literally corresponds to the same effect on the digital objects or a transformation is computed by the system, whether this transformation can be configured over time or not, et cetera.

Figure 7. Koleva et al.'s categorization of Tangible Interfaces [119]

Fishkin [68] proposed a similar analysis on the objects according to two axes: the metaphors and the level of embodiment. Metaphors can be associated to objects in order to link the physical and digital world, while the level of embodiment is generally reflected by how close the system output is to the physical artifact. In particular, for the metaphor classification he built on the previous classification of Underkoffler and Ishii [207], classifying as nouns all the interactions in which the object or a part of the object physically resembles to the digital object and as verbs all the interactions in which the action physically performed on the digital object resembles to the effect generated in the digital world. It identifies then the none metaphor in which the object and the action performed on it have no relationship with the digital object, noun+verb, in which both metaphors are present, and full, in which the object and the interactions are so closely tied to the digital world that cannot be distinguished.

Concerning the levels of embodiment, Fishkin [68] individuated examples in which the output and the state of the system are fully embodied in the objects and examples in which the output is presented nearby the object, in the surrounding environment, or in a distant display. With this classification, Fishkin [68] offers a concrete representation of the concept of embodiment in tangible user interfaces,

which will be better discussed in Section 2.2.2.2. Hoven and Eggen [212] proposed a further classification of objects, extending the previous classification of Ullmer and Ishii [205] with the distinction between objects that have a personal meaning for the user and objects that have not. This classification, where meanings associated to objects are generated by the users, opens a further discussion about the design of tangible interaction that is situated in the everyday environment of the user in which particular context, such as personal experiences and culture, could affect the user interpretation of the tangible interface.

Rather than classifying the physical artifacts that take part to the interaction in TUIs, Hurtienne and Israel [96] proposed a classification of image schemas (Figure 8), i.e., the concepts derived from our previous knowledge that we unconsciously apply when we interact with our surrounding environments. They identified different levels of image schemas in a continuum, according to an increasing degree of specialization: innate image schemas are intuitive reflexes that are encoded in our DNA; sensorimotor image schemas are learned through the exploration of the external world in our early age and are generally common to people worldwide; culture image schemas,

instead, derives from common practices and traditions typical of a particular region; finally, expertise image schemas, are those that human learn by practicing hobbies or by learning new skills for work;

tools are particular image schemas that span across the three latter categories and are developed through the usage of specific artifacts. The spectrum of image schemas is very large (between 30 and 40 according to the authors) and are classified in 8 groups.

Analyzing current abstracting frameworks on tangible interaction, most models are centered on the tangible artifacts used for the interaction, while there are few models that analyze, from a user-centered perspective, the interactions

and in particular the gestures that can be performed with the tangible

Figure 9. Hornecker and Buur's Tangible Interaction framework [95]

investigating the physicality of the gestures and the sensorimotor abilities that could be learnt to interact with tangible artifacts. Also from a semantic point of view, only Fishkin [68] considered the meaning that could be associated to gestures (with the verb metaphor).