The Jazz Improvisation Application Framework

The Jazz Improvisation framework depicted at Table 5.6 shows the same musical information of the GoB application. But the geometric interpretation shows the difference between the two: in the GoB the layers originate from a complete composition which has been sliced in four layers plus the melodic part. Thus the geometric representation is a stack where all the layers have a precise place and whose simultaneous performance generates the complete composition. In the Jazz Improvisation the nine layers do not belong to a complete composition but may be freely superimposed without any claim of completeness. Thus, their geometric representation is a series of interactive landmarks freely arranged on the application’s surface. The user can begin her/his exploration of the interactive landmarks from any accessible point along the borders. However, the possibilities to reach the landmarks depend on the path the user choses, as not all the landmarks can be reached from any point. As depicted in Figure 5.21, the path to activate a precise set of landmarks must be carefully planned by the user who has to find her/his way as if moving in a musical labyrinth. In this case the problem of where to move may have two solutions: to make a random exploration of the environment whatever the musical result, or plan a precise path through the environment in search of a precise musical result. The choice of when to move depends only on the listening time the user needs to understand the music layer s/he is activating and on how s/he wants to use this information. For instance, the user could activate a percussion landmark only for a limited amount of time and than mute it by stepping again. Thus, the timing of the movements depend on the user listening abilities and creativity in designing the

74 Interactive Spaces: Models for Motion-based Music Applications

Figure 5.21: A possible path on the Jazz Improvisation surface. The blue interactive landmarks are the playing ones because they have been touched only once by the user, whereas “eco” and “bass” are muted after the second user’s step.

JAZZ IMPROVISATION APPLICATION FRAMEWORK Aim Music Recomposition

Interface

Musical Information Instrumental layers

Geometric Interpretation Nine free superimposed instrumental layers Spatial Positioning Subdivision of the rectangular active floor

area in nine rectangular interactive landmarks

Interaction

Where to move 1

st

time: begin from any zone 2nd

time on: move to one of the two contiguous positions When to move When the need of adding/muting a music layer is felt

Table 5.6: The table shows Jazz Improvisation aim, musical information, geometrical interpre- tation and spatial positioning of interactive landmarks.

musical environment s/he prefers. As the application’s aim is a free music recomposition based on environment exploration and musical discovery and listening, it can be assigned to the inquiry games category discussed in Section 3.2.3.

Chapter 6

Three-dimensional Spaces Music

Applications

In this Chapter two music applications based on three-dimensional spaces are presented. Both applications deal with interactive music composition and performance expressive interaction. Disembodied Voices employs a user-centered three-dimensional space where the user acts in a hemispherical space in the range of her/his stretched arms. Following the conductor’s model, the user performs impulsive input gestures to progress an interactive algorithmic composition and continuous input gestures to render an expressive performance. “Hand Composer” has similar functions as Disembodied Voices, but employs a smaller sensor-centered space. Both application belong to the category of “no touch” instruments, where the user employs kinesthesia and pro- prioception to direct her/his gestures towards open-space, invisible active regions and thresholds.

6.1

Case Study 4: Disembodied Voices, a Conducting System

Disembodied Voices is an interactive environment designed for an expressive, gesture-based musical performance. The motion sensor Kinect, placed in front of the performer, provides the computer with the three-dimensional space coordinates of the two hands. The application is designed according to the metaphor of the choir director: the performer, through gestures, is able to run a score and to produce a real-time expressive interpretation. The software interprets the gestural data and controls articulated events to be sung and expressively performed by a virtual choir. Hence the name of the application: you follow the conductor’s gestures, hear the voices but do not see any singer. The system also provides a display of motion data, a visualization of the part of the score performed at that time, and a representation of the musical result processed by the compositional algorithms.1

1_{A Kinect Conductor video may be found at https://youtu.be/oyf7GrMMrL8.}

76 Interactive Spaces: Models for Motion-based Music Applications

6.1.1

How it Works

Disembodied voices is a system where a conductor plays a score written for music composition algorithms. According to Mulder (2000, p.316) in the musical tradition there are two forms of musical rendering: the conducting and the performance. The first is a symbolic gesticulation that aims at controlling the musical structures; the second consists in the manipulation of the control surfaces of the musical instruments aiming at an expressive sound production. In Disembodied Voices the conductor performs both functions. The environment is focused on the user’s body positioned at the center of a virtual hemisphere, delimited by a radius corresponding to the length of her/his stretched arms. The user’s right hand movements data produce discrete inputs, or “buttons” (Verplank, 2003), that are employed for the progress of an algorithmic composition score written according to the model of the so-called Ligeti’s “micropolyphony” (Bernard, 1994). The left hand movements produce a continuous data flow, or “handles”, employed for expressive interaction. Thus, the conductor acts on the musical structure creating interactively the timing succession of the different sections of the composition. In the same time, using another gestural repertoire, s/he interprets the score moving on the imaginary control surfaces relative to the interpretative parameters.

6.1.2

Related Work

Interactive conducting dates back to Mathews’s Radio Baton (Mathews, 1991), where control signals were produced employing magnetic capacitance to effect sound production, in a way similar to the Theremin.2 _{In more recent times the Digital Baton (Marrin and Paradiso, 1997)}

developed a similar idea employing a sensorized handheld device capable of detecting beats and movements acceleration for sound expressive interaction. Many other projects like Personal Or- chestra, (Borchers et al., 2004) UBS Virtual Maestro (Nakra et al., 2009) and others, (Toh et al., 2013) and (Maes et al., 2013), were proposed with the main goal to control the musical tempo and sound dynamics. However, all these systems are inspired by a rather traditional idea of the musical conductor, who interacts with the musical result only through the beat and dynamics control. Digital sound processing techniques, as well as algorithmic composition, offer much wider possibilities of controlling music production in real time than the simple beat speed or sound volume. Moreover, Disembodied Voices employs the three-dimensional space all around the conductor as an active space, where imaginary thresholds delimit active regions for digital sound processing. Thus, Disembodied Voices can be considered as a no-touch instrument, be- cause the user acts with her/his hands in the air, crossing imaginary region borders and moving inside them for expressive sound interaction. These are the reasons why the most important Disembodied Voices’s reference is the Theremin. The Theremin uses a simple capacitance mea- surement to sense the proximity of the player’s hand, thus providing the creation of a sensible field that is the instrument’s no touch interface. Also if the sensing techniques employed by the two systems are very different, the playing technique of the Theremin is incredibly close to that employed in Disembodied Voices. The Theremin control space has two dimensions and is completely imaginary. Playing the instrument requires a highly developed kinesthetic sense: it

Chapter 6. Three-dimensional Spaces Music Applications 77

is not only necessary to recognize the position in space (proprioception), but also to have the ability to understand the extent, direction and weight of the movements (kinesthesia). Accord- ing to Billinghurst and Buxton (2011) the success of the Theremin would be right in this direct relationship between hands position in the control space and the continuous sound feedback that allows the player to build his own mental map for playing the instrument. Another no-touch instrument is the Sensor Chair3_{that is interesting precisely because it introduces for the first time}

the concept of a three-dimensional control surface. The Sensor Chair is a device that measures the hands and feet position and motion of a seated occupant. It was developed for MIT Digital Expression Conference in October of 1994 and it has been used as a one of the performance in- struments in Tod Machover’s Brain Opera.4 _{A copper plate affixed to the top of the chair cushion}

is a transmitting antenna being driven at roughly 70 kHz. When a person is seated in the chair, s/he effectively becomes an extension of this antenna; her/his body acts as a conductor which is capacitively coupled with the transmitter plate. Four sensors provide the xy plane position for the hand as well as z position of the hand’s distance from the sensor plane. This coordinates data have been used to launch a sound and adjust its volume (xy) and to change its timbral charac- teristics (z), or to divide the xy plane into many zones which contain different sounds (Paradiso and Gershenfeld, 1996).