Reality-based Interaction Themes in Interactive Spaces

Jacob et al. (2008) identify also some physical world themes which help to understand how reality forges interaction in post-WIMP interfaces. These themes are the following:

36 Interactive Spaces: Models for Motion-based Music Applications

Figure 4.3: Three different sizes for digital windows in interactive spaces. Form the left a context-aware device, a tabletop and a large resolution display of the AffinityTable pro- totype (Jetter et al., 2014), and the responsive floor of Google’s Interactive Spaces (source arstechnica.com).

friction, gravity and so on (e.g., scrolling through a menu that simulates the resistance as if it were a spring, or widgets animations with mass and stiffness simulations);

• Body awareness and skills that take in account the perception users have of their body in space, e.g., proprioception (the sense of the body and limbs position) and peripersonal space perception;1

• Environment awareness and skillsthat relate to spatial cognition that users employ when moving around the world in their everyday life;

• Social awareness and skillsare the abilities to relate to others and comprehend local co- presence, social affordances and sense of social presence.

These themes are employed to understand the analogy between real-life events and the augmented- reality experience in an interactive space. Particularly, they are reinterpreted in order to provide a basis for the construction of the conceptual framework for motion-based music applications presented below.

4.1.2.1 Na¨ıve physics

The na¨ıve physics examples presented above relate to the digital representation of objects be- havior in the natural environment. In an interactive space the user’s body is at the origin of the interaction and, consequently, real-world movements of one or more bodies and limbs are the focus. The characteristics of these movements may express data that can be used to produce some augmented reality effect like sounds or interactive projections, as depicted in Figure 4.4. Movement qualities expressed in the Labanotation2 like effort factors (space, weight, time and flow), can help the understand the expressive potentialities of these data. Some examples can

1_{The peripersonal space is the space around the body and in the reach of the limbs.}

2_{Laban Movement Analysis, LMA is a method for describing and categorizing human movement. It was pro-}

Chapter 4. A Conceptual Framework for Motion-based Music Applications 37

Figure 4.4: An interactive dance setup, with projections depending on dancer’s movement. Source http://www.thisiscolossal.com

be found in Mappe per Affetti Erranti, a contemporary dance project based on the already de- scribed Eyesweb XMI Library (see Section 2.3.1.2). In this project twelve expressive descriptors (quantity of motion, impulsiveness, vertical and horizontal components of velocity of peripheral upper parts of the body, directness index, etc.) are employed to define four expressive intentions that control the musical output (Camurri et al., 2008). Thus, the physics of full-body movements becomes the main interaction modality in an interactive space and the quality and richness of this interaction is particularly interesting for musical environments where expressivity plays an important role.

4.1.2.2 Body awareness and skills

As the user of an interactive space affects the environment through her/his movements, body awareness and skills play a fundamental role in the interaction. One of the skills users employ in their everyday life is embodied cognition, which expresses aspects of user’s experience that are grounded in the physical body and sensorimotor memory. Relevant psychological studies in the field of cognitive and pedagogical sciences, like Fodor (1983), Clark (1998) and Piaget et al. (1952), emphasize the importance of sensorimotor information as part of cognitive processes fed by body experiences in the world. Leman (2008) in his studies emphasizes the role of embodied music cognition as an effective mediator between music knowledge and technology. Thus, not only there is no separation between the abstract and the embodied mind but, as George Lakoff and Mark Johnson affirm, many abstract concepts can be understood through image schemas which are stored in our sensorimotor memory and which come from bodily interactions (Lakoff and Johnson, 2008). Some of these schemas can properly describe some important spatial char- acteristics of abstract concepts, and, in particular, they can be adapted to depict musical features. For instance the source-path-goal schema expresses the quality of melody proceeding by orga- nizing the experience of goal-directed motion; or the container schema can very well represent the concept of boundary regions like that of tonal harmony (Brower, 2000). More simply, one

38 Interactive Spaces: Models for Motion-based Music Applications

of the most elementary musical structures, the scale, is commonly described with terms like “as- cending” and “descending”, which are body-movement and spatial metaphors. In experts as well as in common-people language melodies “go up or down” or “jump” or “arrive” to a particular point. Tones can proceed by “skip” or by “step”, keys can be “near” or “far”, harmonies make “turnarounds” and bass lines are sometimes defined as “walking basses”. All these examples show how musical abstract concepts are deeply grounded in users sensorimotor information and, at the same time, introduce the theme of spatial cognition as linked to embodied cognition. 4.1.2.3 Environment awareness and skills

Environment awareness and skills refer to how a user perceives and manages environmental elements functions and spatial organization. This knowledge is composed of spatial properties like location, size, distance, direction, separation, connection, landmarks and route information, and is acquired via sensorimotor systems that operate as people move about the world. Also the ability to detect environmental acoustic and non-acoustic events is fundamental to coordinate actions according to ecological occurrences (Phillips-Silver et al., 2010).

4.1.2.4 Social awareness and skills

Interactive boards or large touch-screens allow to extend the traditional WIMP interaction to more than one user. When the interactive space is so large to be not only visible to a great number of bystanders but also to be acted by many users, the users actions acquire a completely different meaning. The social sharing of the interactive experience and co-located users participation to the reactive environment exploration makes the consequent cognitive processes a transparent, public event. This may lead to various outcomes as social facilitation,evaluation apprehension, audience enjoyment and emotion arousal enhancement (De Kort and Ijsselsteijn, 2008).