Discussion of Results

The results paint a picture of a relatively complex interface that works on three levels, re- quiring some level of skill in each. Firstly, there is the functional calibration stage, at which the user needs a level of competence to establish a basic level of functionality. Following this there is training and then use, each involving different kinds of creativity, firstly in defining a language of interaction, and then in using this language to interact musically. The creation of a successful language is fundamental in a system such as this, and the key issue here is estab- lishing a meaningful connection between motion and audio output. The importance of this relationship is explored by Antle et al. (2009), who found that use of an embodied metaphor could make a system more intuitive to use. This is echoed in the evaluation results; one par- ticipant had difficulty making a symbolic connection between motion and sound when using the system, and commented on how the extra cognitive load of using a new vocabulary of ges- tures made the system more difficult to use. Another participant, who imagined metaphors when interacting with the system, found the system intuitive and natural. In Phalanger, this connection between motion and sound is established over a two level mapping process, first by defining gestures, and then by defining mappings between the system output and the audio

engine. The conceptual split between these two stages may contribute to some difficulties in use of the system.

Returning to the wider research theme, the results highlighted some issues surround the musicality of the interface. One participant enjoyed the potential for error in the interface, feeling it contributed to an instrument-like feel. This theme of imprecision and unpredictabil- ity is also observed in other studies of computer musicians (Bertelsen et al., 2007; Magnusson and Mendieta, 2007; Gelineck and Serafin, 2009b), and this highlights the importance of in- corporating these factors into musical interaction design. Participants commented on the expressive potential of hand use; a system such as this has an inherent convergent mapping both on a physical and logical level, translating from the many degrees of freedom of hand mo- tion into a lower dimensional set of continuous parameters. Wanderley and Depalle (2004) suggested that convergent mappings have a high potential for musical expressivity, and this is confirmed in the results. The interplay between this high degree of freedom and potential of error underpins the intangible style of interaction in Phalanger.

5.4 Summary

A system has been presented that enables musical control with hand motion, achieved by tracking the hand with a combination of computer vision and machine learning techniques. A formative, laboratory based evaluation revealed potential improvements to the system; these were implemented and the system evaluated in more depth in a qualitative field study. The re- sults showed that hand motion tracked in this manner can be successful in creating expressive, musical interaction when there is a meaningful relationship between hand motion and sound. The creation of this relationship presented some potential difficulties, and questions remain open in the area of how to design a language of interaction, and how this relates to creative workflow, cognitive style (Eaglestone et al., 2008) and the musical practice of the user.

The Phalanger system was explored in a further study, as an interface for a timbre space exploration system. This work is presented in chapter 7. The study elicited more data about user experience with this system. Along with the formative and field study results, this gives a total of three sets of results from which to triangulate an understanding of Phalanger. These results shall be explored together in chapter 8.

Echofoam: A Malleable Controller

‘It’s got to the point now where my work is really about riding that knife-edge between what works and doesn’t work, absolute control and no control. Disaster and delight. Life, really!’

6.1 Introduction

The EchoFoam system is a new controller that further explores the nature of embodied con- trol of digital music tools, following a multiparametric paradigm. While Phalanger explored freeform hand motion and intangible interaction, EchoFoam looks at different aspects of motion capture, using a tangible interface. The system follows a paradigm of manipulating malleable material in order to afford detailed and intuitive control.

Malleable interfaces are currently seeing commercial outings in the form of the SUMA1

and Blobo2 _{devices, both marketed as squeezable controllers. There has also been plenty of}

activity in the academic sphere. Schwesig (2008) describe the concept of Organic Interfaces, sensitive analogue devices that acknowledge the subtleties of physical interaction; they illus- trate their ideas with the hypothetical Gummi device, a deformable display that responds to physical manipulation by the user, arguing that subtle physical interaction with a real-world object such as this would lead to a suspension of disbelief, a quality perhaps also desirable in a musical controller. Moving to real-world examples, Reed (2009) created a prototype digital clay, using embedded wireless sensors and computer vision to measure manipulation of the material. Sato, Mamiya, Koike, and Fukuchi (2009) also use computer vision techniques in their PhotoelasticTouch system, which measures the deformation in transparent rubber objects in a tabletop interface. An example very relevant to this project is Smith et. al.’s work; they created several input devices using configurations of multiple conductive foam sensors, for use as interfaces for 3D sculpting and camera control (Smith, Thomas, and Piekarski, 2008b,a). Milczynski, Hermann, Bovermann, and Ritter (2006) created the Elastable, a device that em- ploys computer vision to measure deformation of a rubber surface in order to explore and sonify high-dimensional data sets .

Focusing on musical examples, a novel route to malleable control is taken by Hook, Tay- lor, Butler, Villar, and Izadi (2009). They used an array of ferromagnetic sensors to measure deformation in a ferrofluid filled bladder, in one example mapping this to synthesis parame- ters in Max/MSP. Chang and Ishii (2007) designed the ZStretch musical controller, a fabric device that measures deformation using resistive strain transducers sewn into lycra. Marier (2010) embedded accelerometers and force sensing resistors into a piece of sponge to create a malleable musical controller. Lastly, another musical example is Weinberg and Gan’s (2001)

Squeezables. They embedded pressure sensors into several soft gel balls which were played to-

gether as a collaborative instrument. An evaluation study showed that the players valued the expressiveness of this style of interaction.

The system described here uses a malleable foam sensor, tightly coupled with reservoir computing mapping techniques, to create a device that can measure subtle physical manip- ulations and map them to multi-dimensional control streams. It is used in this case for the haptic exploration (Lederman and Klatzky, 1987) of sound synthesis parameter spaces. Two user studies, one lab-based and one field-based, evaluate the success of this system and high- light some interesting issues surrounding malleable control.

1http://www.cambridgeconsultants.com/news_pr257.html 2http://www.bloboshop.com/