Multi-touch interfaces

Multi-touch devices have evolved and disseminated incredibly fast, due mainly to the wide range of uses in several areas, but with the most impact in mobile communi- cations, where the advantages of the versatility and simplicity of use of the virtual screen interfaces practically took completely over the existing hardware counterparts. The virtual interface eliminated or significantly reduced the space and hardware inter- face limitations of the mobile telephones thus allowing them to become much more than telephones and smartphones with basic and mostly unusable extra features. The iPhone, introduced by Apple in June 2007 (“Apple product timeline,” n.d.) redefined the paradigm of the mobile phone as a new line of mobile mini-computers, with features that were really usable. In fact, this new line of devices was so effective, that the con- cept was reaffirmed with the introduction of the iPod Touch in September 2007, and the iPad, in April 2010.

Along with the devices, the key to this success was the opening of their operating system (iOS) to third party developers, thus creating a very attractive platform both for

users and developers, deploying their software through the App Store. Since its intro- duction in 2008, Apple’s iOS App Store has seen an amazing growth on the number and quality of applications of all types, exceeding every prediction. Music apps were no exception, and very soon showed up, using the iOS technical possibilities as well as exploring new possibilities for multi-touch control and mobility.

Different types of music apps can be found. Here I will enumerate some of the more relevant ones, related to music playing and music production, namely:

- Virtual Instruments;

- Music Production applications; - Music Controllers;

Other types of music related apps exist, like music players, browsers, song data- bases and recognition algorithms, music notation, and more. These are, however, not so relevant for the current dissertation and as such will not be included here.

Virtual Instruments

Virtual instrument applications use the device as a musical instrument, by providing some kind of control inputs that convert the user’s physical actions into musical events. A distinction can be made between a) virtual instruments that simulate existing hard- ware musical instruments, whether acoustic or electronic; b) virtual instruments that simulate existing music software; and c) virtual instruments that present original inter- faces and modes of interaction.

Applications in the a) and b) categories try to emulate as close as possible the experi- ence of playing the corresponding original instrument or software, frequently using pho- torealistic graphical interfaces and the devices features like the accelerometer or the

microphone. In Ocarina (Smule, 2013), the user can blow into the microphone in order to simulate blowing into a real ocarina, while pressing the virtual holes on the touchscreen for note selection. Guitarist (MooCowMusic, 2013), Virtual Guitar (NETTuno s. r. l., 2013) and Real Piano (Cookie Apps Inc., 2013) are examples of acoustic instrument simulation, while Korg MS-20 (Korg inc., 2013) and Reactable Mo- bile (ReacTable SystemsSL, 2013), for example, are very well known examples of simu- lations of existing hardware.

The real-life simulation approach, however, is not necessarily the best way to go re- garding the use of the screen and multi-touch capabilities. In fact, these virtual instru- ments can be quite difficult and even ineffective when getting down to actually using them, whether because of limitations of the screen space, response latency or simply because of the lack of the tactile sensation and physical feedback of the real instru- ment. Apps like Bloom (Opal Limited, 2013), SoundPrism (Audanika, 2013), Nodebeat (Affinity Blue, 2013), Thumbjam (Sonosaurus LLC, 2013) or Orphion (Trump, 2013), for example, have a different approach by presenting original instruments and interfaces which can be more interesting and effective to play and to explore new musical possi- bilities. Others, like the impressive Animoog (Moog Music Inc., 2013), combine both approaches to allow very expressive performance while retaining some of the visual and organization paradigms.

Music Production

Music production apps include sequencers, audio recording and editing tools, and virtual studios, which can have much functionality and can put together several types of instruments and/or effect processors. Most of these apps have also a simulation ap-

proach, by porting the experience of existing computer software to the mobile device. Apps like MusicStudio (Gross, 2013) and the very impressive Garage Band (Apple Inc., 2013) are very good examples of this, while also using some new features exploring for example the multi-touch properties of the iOS devices. Others, like Tabletop (Retronyms, 2013) or Rhythm Studio (Pulse Code Inc., 2013) provide the user the ex- perience of having a modular set of devices in a virtual tabletop.

Music Controllers

This type of applications allows the use of the iOS device as a controller for external software or hardware. Some apps like for example AC-7 (Saitara Software, 2013) and ProRemote (Far Out Labs, 2013) are DAW (Digital Audio Workstation) controllers, which work like remote wireless controllers for a computer running the corresponding DAW software. Some well-known DAWs have nowadays their own release of remote control- lers for iOS, including MOTU’s DP Control (Motu inc., 2013) and Steinberg’s Cubase iC (Steinberg, 2013).

Some more open apps like Touch OSC (Fischer, 2013), MrMr (10base-t interactive, 2013) or Fantastick (Pink Twins, 2013) allow the user to build his own layout and pro- gram the OSC or MIDI messages for every item. In fact, these can be used for any oth- er non-music software that accepts configurable external control messages, like real- time graphics or video applications.

2.5. Conclusion

The topics and examples addressed in this chapter provided a base for some im- portant aspects that were considered in the development of the concepts and solu-

tions devised for the algorithms and software created for this dissertation. In this case, the selected topics inform about the main considerations in the definition of the output format of the software, which are both conceptual and technological.

The concept of musical interface is key to the proposed model of a computer- mediated performance system that I will describe in the second part of this document. The main concepts addressed in this chapter include a reflection about musical inter- faces, the definition of instrument and different instrument formats, in order to reach my own definition of instrument, that, hopefully, is simultaneously broad and encompass- ing, but also explicit and specific enough to serve the objective and purpose of this dis- sertation.

The notion of “meta-instrument” is one of the core concepts of this dissertation. The control of musical processes at a meta-level has been around since very early in com- puter music history and the computation power and the technological resources avail- able today renders possible an endless number of possibilities for the creation of musi- cal systems empowered by sophisticated algorithms, and with any combination of in- put and output formats. Using specifically developed mapping and computation algo- rithms, the meta-instrument can provide a reference for the development of new for- mats of musical interfaces and software.

The provided examples of commercial software are important to emphasize how mainstream software companies are gradually approaching and integrating features obtained with computer mediation.

The provided sample list of musical applications illustrates the new range of possi- bilities in the music hardware and software development business, and that the multi- touch tablets are undoubtedly a powerful and attractive platform for musical use, re-

sponding to some important features that were always desired in digital musical in the digital domain, namely power, versatility, mobility and interface flexibility and intuitive- ness. However, they are still limited by the lack of a tactile sensation and physical feed- back, but this may not necessarily relevant in many cases.