Spatialization – Hidden Lines

6.5 Spatialization – Hidden Lines

An improvisation experience for electroacoustic instruments playing over automatically- preprogrammed trajectories in a multichannel speaker setup.37

6.5.1 Tags

Spatialization, ambisonics, VBAP, amplitude panning, multichannel, envelopes.

6.5.2 Goal

6.5.3 Description

Improvisation requires a multichannel loudspeaker set up and electroacoustic sound sources, preferably with mono outputs. Though doable, handling stereo or more outputs per instrument increases complexity, blurring the clarity and purpose of the exercise. The speakers can be arranged in different configurations, for example:

• Ring of similar loudspeakers (4-8) surrounding the performers with a possible audience in the center.

• Hemispherical dome-like (half of a sphere) arrangement, with performers in- side the geodesic structure using similar characteristics of loudspeakers. • loudspeaker orchestra with a variety of loudspeakers arranged in an orchestra-

like disposition. Mostly frontal but sides, highs, and back can be addressed according to the available units.

• Tree-like structure with loudspeakers arranged with a common center but pointing to different directions in the room. Performers are placed surround- ing the structure, and a hypothetical audience sits on the periphery.

• One or two speakers per performer placed behind them arranged in a line or slightly curved arc. An audience will ideally be placed in front of the band. • Symetrical or asymetrical distribution of loudspeakers among the audience.

37_{This chapter is strongly inspired by Dr. Andrew Bentley’s courses on advanced tech-}

niques of spatialization, and some of my colleagues’ work in the doctorate program in music technology (Dom Schlienger, Alejandro Montes de Oca, Juan de Dios Mag- daleno and Paola Livorsi)

One computer running a spatialization software, e.g. VBAP,38_{Ambisonics Toolkit,}39

Spat,40 _ReaSurround,41 _{or similar. The computer must handle all the loud-}

speakers through a sound card with enough analog channels as required. All the performers connect to the same sound card for spatialization.

A series of patterns controlling the parameters of the spatialization software to create trajectories must be programmed in advance. Fixed-point positions must also be included. Each spatial movement can have a fixed, determined duration, or it can be calculated and renewed on each iteration. This duration will reflect the speed of the trajectory in proportion with the number of loudspeakers. It should be important to keep track of the duration length of each trajectory to allow control over the time extent of the whole exercise.

The exercise starts with silence and evolves into a free improvisation. The com- puter running the spatialization software will algorithmically assign to each of the inputs a trajectory with a number of repetitions. When this initial trajectory has played in its totality its repetitions, a new pattern will be chosen until it reaches either a natural end or a pre-programmed total duration. The performers may react or integrate the spatial trajectory into sonic gestures and improvisation. For example, by reflecting the spatial trajectory: if a circular movement is proposed, the music can suggest periodicity or circularity. Contrasting and divergent sonic gestures are another alternative for the performers to react to the spatial lines. For example, slow random movements can induce a continuous musical gesture. Many other creative answers can be found on the spot of the improvisation for reacting to the sound put in motion on the space.

The choice of the trajectories and the number of repetitions and durations of each pattern assigned to each performer can be set to random, giving the possibility to each of the paths to appear at any moment. Another option is to program an overall form with convergent tendencies based on probabilities, for example, starting with slow and simple recognizable movements like a circle, then progres- sively increasing the speed and complexity of the patterns and eventually coming back to a similar state as in the beginning, or any other overall form. This type of form can be achieved by giving weighted possibilities, so some durations and pat- terns may have a higher possibility to appear than others following a set of rules. The main point of the exercise is to put the performers in a situation where they

38_{Pulkki, Ville. Spatial Sound Generation and Perception by Amplitude Panning Tech-}

niques. Helsinki University of Technology, 2001.

39_{Heller, Aaron J., Eric Benjamin, and Richard Lee. “A Toolkit for the Design of}

Ambisonic Decoders.” In Linux Audio Conference, 1–12, 2012.

40_{Carpentier, Thibaut, Markus Noisternig, and Olivier Warusfel. “Twenty Years of}

Ircam Spat: Looking Back, Looking Forward.” In 41st International Computer Music Conference (ICMC), 270–277, 2015.

41_{Néron Baribeau, Raphaël. “Méthodes de Spatialisation Sonore et Intégration Dans}

6.5 Spatialization – Hidden Lines 177

have to attentively listen to the movements of their sounds in space and musically react to that.

6.5.4 Variations

An important aspect of the performance experiment proposed here is the listen- ing conditions of the performers. Therefore, the position in the room of both performers and loudspeakers have to be carefully chosen. While listening in the same space as the audience may facilitate the technical setting, it makes it dif- ficult to assure an optimal listening point for everybody. One possible variation to handle this is to use a monitoring system with headphones running a bin- aural rendering of the mix. Setting up such a system requires a good binaural encoder/decoder of the room or a dummy head placed on the sweet spot and redirecting that signal to each performer.

An exciting approach to sound spatialization is to physically move the loud- speakers, either by a performer or by an electro-mechanical system. I suggest experimenting with wired or wireless speakers moving in the room, coupling a performer producing sounds and another one playing the spatialization. Small electromechanical machines can also be built around step or dc motors and mounting or attaching the speakers to them, alternatively, modifying turntables or any other device that can put in motion or turn a loudspeaker.

6.5.5 Discussion

There are two important components in this unit: the spatialization of sounds and the pre-programmed trajectories.

Sound spatialization refers to the use of loudspeakers to create a spatial, musi- cal experience.42 _{Since most of electroacoustic instruments can integrate a loud-}

speaker or a similar transducer to transform electric signals into acoustic energy, delivering sounds in a physical space, sound spatialization can be placed at the heart of the learning process in electroacoustic music. With the exception of some instruments that may have a built-in speaker, the property of electroacoustic in- struments of decoupling the sound generator from its diffusion system has opened up research and created an incredible amount of implementations, questions, and even musical trends to learn from, to explore and to investigate.43

42_{"Spatialisation, Rob Weale" accessed June 2018,}

https://web.archive.org/web/20181108030034/http://www.ears.dmu.ac.uk/

43_{Baalman, Marije A. J. “Spatial Composition Techniques and Sound Spatialisation}

From the simple form spatialization using a single loudspeaker, a lot of experi- mentation and learning can be carried out. The position, direction, elevation, and angle of the loudspeaker will have strong consequences in the perceived sound and can easily be tested, for example, by fixing or freezing all the other sound parame- ters, e.g. a test signal (ultimately, other than noise and sine tones) like a sonic ges- ture or a musical sentence, played again and again while affecting the positioning of the loudspeaker. It should become obvious how interdependent the interaction between the physical properties of the space and the loudspeaker are. Playing the loudspeaker becomes playing the space or even more playing the room. After experimenting with position and therefore movement of the loudspeaker, a new dimension can be introduced by acoustically manipulating the sound emitted, for example, by using diffusing materials in front of the speaker elements and me- chanically interacting with them and even beyond that of course, experimenting with transducers and different surfaces/objects for sound diffusion.44

It is probably not easy to imagine the number of questions that are introduced by increasing the number of loudspeakers to two. The whole field of stereophonic sound enters the game. How are the sound signals distributed over the two speak- ers, i.e., mixing and panning techniques, how the position, elevation, angle, dis- tance between the loudspeakers and the listening point interact, how to produce a perception of movement in the sound source and ultimately how to perform the stereophonic space? From the performance perspective, playing within a stereo- phonic system has musical and sonic consequences that must be addressed and carefully studied. The interface to control and manipulate the perception of the stereo field is something to research by each electroacoustic musicians; from the panning pot paradigm to the multitouch controllers, an array of interfaces and available solutions implementing different technics are available.45

Multichannel systems: quadraphonic, octaphonic, 5.1, 7.1, 9.1, and other surround systems require for the performer a focus on the organization of the sounds in space that may feel overwhelming when handled simultaneously with the sound production task. Therefore, it is common to see this task delegated to a dedicated performer controlling the diffusion system, either with a traditional mixing desk or a dedicated controller and dedicated software.46

44_{Lähdeoja, Otso. Composing for an Orchestra of Sonic Objects: The Shake-}

Ousmonium Project. Ann Arbor, MI: Michigan Publishing, University of Michigan Library, 2016.

45_{Madden, Andrew, Pia Blumental, Areti Andreopoulou, Braxton Boren, Shengfeng}

Hu, Zhengshan Shi, and Agnieszka Roginska. “Multi-Touch Room Expansion Controller for Real-Time Acoustic Gestures.” Audio Engineering Society, 2011. http://www.aes.org/e-lib/browse.cfm?elib=16102.

46_{Pysiewicz, Andreas, and Stefan Weinzierl. “Instruments for Spatial Sound Control}

in Real Time Music Performances. A Review.” In Musical Instruments in the 21st Century, 273–296. Springer, 2017

6.5 Spatialization – Hidden Lines 179

As an alternative to having a space performer, an automatic system can be consid- ered. Morton Subotnick developed during the seventies a technique that he calls Ghost Scores and Ghost Electronics.47_{In this technique, Subotnick pre-recorded}

control data on quarter-inch magnetic tape to derive voltage control from it and drive audio processors on live instruments. In this way Subotnick could achieve a precise control of the electronics over entire compositions. The idea of sepa- rating the control data into a pre-recorded form making it an automatic process obviates the need for a dedicated performer. In the experiment proposed here, that data will be the modulation of spatialization parameters over time, creating sound localization and motion effects. One option to carry out the modulation consists of using envelopes. An envelope is built around segments with target val- ues, duration time and curve types. When triggered, the envelope will modulate the parameter over the duration determined by each of the segments, for example, going from value 0 to 1 in one second in a linear progression and staying at value 1 for two seconds and returning to 0 in 4 seconds in an exponential sequence. Each of these values can be pre-programmed by hard-coding them and creating a library or using stochastic methods of probability.

Each spatialization software implementation has been exposed to the user control parameters. For example, as discussed by Marshall et al. in the paper “Gesture Control of Sound Spatialization for Live Musical Performance”, some of these control parameters include: position (x,y,z), azimuth, elevation, size, directivity, presence, distance, brilliance, reflections, reverberations, doppler effect, equaliza- tion, air absorption, decay.48 _{To control these parameters with the above de-}

scribed envelopes, it will be important to normalize or to map the range of the parameters in a known scale, for example, 0 to 1 (or 0 to 127 if using MIDI as the control protocol). Then a connection between the control data (the envelopes) and the parameters have to respond to a perceptible effect of motion, direction or localization on space. For these purposes, each algorithm has to be tested and studied. In some cases it will be necessary to affect only one parameter, while in others, one controlling envelope should affect several parameters simultaneously, and in other cases, each parameter may require its own modulating envelope to achieve the desired effect.

The triggering of the modulation parameters can be chained as suggested above, i.e. once the envelope is over, a new one will be triggered or it can be programmed with conditionals. Conditionals act as control units, performing logical operations between two or more inputs. For example, if one input is bigger than the other one, then output a trigger; otherwise, output nothing. Using conditionals can give

47_{Hanson, Jeffrey. “Morton Subotnick’s Ghost Scores: Interaction and Performance with}

Music Technology”. MA thesis. 2010.

48_{Marshall, Mark T., Joseph Malloch, and Marcelo M. Wanderley. “Gesture Control}

of Sound Spatialization for Live Musical Performance.” In Gesture-Based Human- Computer Interaction and Simulation, 227–38. Lecture Notes in Computer Science. Springer, Berlin, Heidelberg, 2007.

to the performer the possibility to interact with the automatization system. A classical implementation–discussed in chapter 6.3–will be to follow the amplitude of the incoming audio signal and using a threshold value as comparison to trigger the next envelope. In this implementation, each new event of spatialization will be dependent on peaks or silences in the audio stream of the performer. The system will stop listening the incoming audio until the envelope has played in totality. Other mapping strategies can be explored, for example, dynamically changing the duration of the segments in the envelope according to the density of the spectrum, or following the number of events per second in an audio stream and coupling this value to the target values in the envelopes.

With these examples it should become clear how important the mapping strate- gies are between the control data and the spatialization parameters. A reasonable amount of time has then to be allocated to investigate the control of parameters of the algorithm used for the spatialization, a fine tuning of the physical position of loudspeakers, and the choice of the monitoring system for the performers. Con- sequently, the actual preparation for this activity can go well beyond the actual improvisation session.

6.5.6 Implementations

† Program an algorithm to generate envelopes of different characteristics: num- ber of segments, type of curves, durations that can be mapped onto a spa- tialization software via MIDI or OSC.

† Create a pool or database of unipolar waveforms that can be used as control data.

† Study, build or experiment with a voltage control quadraphonic/stereo mixing module.

† With a variety of loudspeakers and amplifiers, set up a lo-fi loudspeaker or- chestra that can be easily transported. As an inspiration, look at the work of Michael J. Schumacher and his multichannel portable system.49

† Map a controller MIDI, OSC or custom made for manual control of the spa- tialization software.

† Research the concepts, ideas and implementations of Ambisonics, Vector Base Amplitude Panning, Wave Field Synthesis.

49_{“Michael J. Schumacher — Portable Multi-Channel Sound System,” August 18,}

2018.

https://web.archive.org/web/20180818122208/http://michaeljschumacher. com/PM-CSS.