iOS-Csound API

The iOS-Csound API was developed by Steven Li and Victor Lazzarini and released in April 2012 [Lazzarini, 2012]. The API allows for the audio pro- gramming language ‘Csound’ to run in the iOS environment, and for the two to communicate. iOS-Csound works in a very similar way to libpd; iOS and Csound can simply communicate data back and forth. This allows each system to complement the other; Csound dealing with the complex audio processing, and iOS dealing with the user interaction.The Csound-iOS API is available for download from the following link:

2.9 Alternative Audio Processing Environments for iOS

2.9.3.1 Csound Overview

“Csound is an incredibly powerful and versatile software synthesis program. Drawing from a toolkit of over 450 signal processing modules, one can use Csound to model virtually any commercial synthesizer or multi-effects pro- cessor.” [Boulanger, 2001]

Csound is a text-based programming language, written in C. It was written by Barry Vercoe in the late eighties, and has developed into one of the major audio-based programming languages. It is able to use hundreds of inbuilt, and third party functions called opcodes to produce a variety of sounds and effects, making it a highly versatile audio processing environment. Though Csound is not as interactive as many other audio based programming lan- guages, such as Max and Pd, it allows for communication with numerous external devices and software. Csound can be downloaded from:

http://www.csounds.com

2.9.3.2 Evaluation of Csound-iOS API

The Csound-iOS API is a highly promising alternative to using Core Audio. An examples folder that comes with the API outlines its key features, show- ing how it is possible to embed Csound as a highly efficient audio engine while reaping the benefits of the iOS user interaction experience. Three significant applications made using this API are outlined below:

• csGrain – A real-time audio processing tool for iPad, that can take sound input and process it using granular synthesis. [Boulanger, 2012] • Dandy – a simple sampler for iPhone. [Hepper, 2013]

2.9 Alternative Audio Processing Environments for iOS

• Density Pulsaret – an extensive granular synthesis engine for iPhone and iPad. [Petrolati, 2013]

These applications demonstrate the processing of the Csound audio engine in the iOS environment. Though mostly focused on granular synthesis, these applications suggest that Csound-iOS is capable of very high performance audio processing with no noticeable latency.

3

Relevant Projects in the Area

As there has been little prior work on this precise area of research it is im- portant to break it down, and study its constituent parts. When combining any number of techniques, or technologies, it is essential that every part is well studied. This survey of the previous work has been done such that it is possible to add to the work of others and expand the knowledge in the area, but still ensure the project maintains some novelty. This chapter covers the following areas:

• Interactive Sonification for Task Monitoring (3.1) • Sonification of Graphical Data (3.2)

• Spatial Auditory Display (3.3)

The work done is then discussed with an aim of exploring what data may be of use, and what information can be applied to this project.

3.1

Interactive Sonification for Task Monitoring

Interactive sonification is a wide varying field, which has led to a great diver- sity in the projects and applications associated with it. Notably, it has been used to provide real-time auditory feedback to allow for the improvement of tasks. For example, it has been used to improve the analysis of EMG (Elec- tromyography Data) [Pauletto and Hunt, 2009], the generation of games for the blind [Papetti et al., 2008], and has provided insight into how a person moves, in the context of physiotherapy [Vogt et al., 2009].

By providing an extra modality, interactive sonification can allow for the user to monitor a task they are undertaking, and improve their performance,

3.1 Interactive Sonification for Task Monitoring

all while remaining eyes-free. An example of this is Schaffert et al’s [Schaf- fert and Mattes, 2012] approach to using auditory feedback to keep visually impaired, and non-visually impaired rowers in synchronisation using sound. This was done by developing a system called ‘sofirow’ (Figure 3.1), which transforms the propulsive acceleration of a boat into sound – mapping the information directly to the tones on a MIDI-scale (between 0 and 127).

Figure 3.1: The sofirow system being used – from [Schaffert and Mattes, 2012]

In general, the users found the mapping aesthetically poor, stating that the sound was “irritating and confusing”. Initially this put the users off, but given some time to learn the system they found it less irritating, and that it helped them improve their overall performance. It is clear that when design- ing a sonification system, that the aesthetics of the proposed sounds should be tested on users beforehand to ensure that they are not irritating with extensive use.

However, it must be noted that the aesthetics of the sound should not be the only focus of the sound design in an interactive sonification system. In [Vogt et al., 2009] it is evident that the quality of the information provided was compromised for aesthetics. Vogt et al. describe an approach for interactive sonification for rehabilitation in which motion tracking was used to convey

3.1 Interactive Sonification for Task Monitoring

vital information about a user’s body shape – rewarding them with positive auditory feedback for performing the correct motions, and giving them neg- ative auditory feedback for deviating from them.

Vogt et al.’s system, depicted in Figure 3.2, uses sound mappings such as ‘rustling leaves’ or ‘animal sounds’. It is assumed that sounds such as these were chosen for their aesthetic value – sampled sounds are generally less harsh to listen to than the aforementioned MIDI scale [Schaffert and Mattes, 2012], and other more pure-tone based approaches such as [Pauletto and Hunt, 2009] and [Meijer, 2013]. However, removing significant amounts of information by ‘filtering’ the data into categories beforehand may not only mean that some vital data may be lost in the process, but also that the sound mappings may be unintuitive, for example, we may not necessarily associate ‘wind in trees’ with ‘up’, as proposed by Vogt et al. [Vogt et al., 2009]. It is essential that there is a trade-off between providing all of the data, and filtering it for aesthetics or ease of use. This is something that is particularly relevant in the next section (Section 3.2)

Figure 3.2: Metaphor based sonification of arm movement – from [Vogt et al., 2009]

3.1 Interactive Sonification for Task Monitoring

As well as monitoring tasks in the real world, some researchers have worked towards giving users feedback about the way they are interacting in a com- puter interface with sound. As mentioned in Section 2.3, auditory icons [Buxton et al., 1994] and earcons [Blattner et al., 1989] can be used to pro- vide some feedback to a user about their interaction with a computer. More recently, work such as [Fernstr¨om et al., 2004] describe approaches to human- computer interaction that use interactive sonification to allow users to com- plete ‘eyes free’ tasks on an non-tactile interface. Fernstr¨om et al. describe a method for allowing a user to perceive ‘pseudo-haptic’ buttons from sound alone. This was done by trying to emulate the sound that friction would make by mapping a user’s touch pressure, and velocity to a sound engine that emulated the frequency characteristics of surfaces when touched. They note the importance of low latency between interaction and feedback as our auditory systems perceive temporal information significantly faster than our visual systems [Fernstr¨om et al., 2004].

Figure 3.3: Haptic vs Pseudohaptic sketch accuracy – from [Fernstr¨om et al., 2004]

By focusing on an exploratory approach [Gibson, 1961] to getting the gist of an interface, Fernstr¨om et al. report some interesting findings. The results