A B S TR A C T
Phonological and Musical Loops
in Live Coding Performance Practice
D Av i D K i M - B o y L e
In 1974, cognitive psychologists Alan Baddeley and Graham Hitch proposed a model of working memory, the fundamen-tal principles of which have received considerable evidential support through wide-ranging experimental observation [1]. The original model consisted of two components, the
visuo-spatial sketchpad and the phonological loop, responsible for
the processing and encoding of information in the visual-spatial and speech domains respectively, each of which work under the direction of a third component, a central executive function. The model was extended by Baddeley in 2000 to include an episodic buffer that helped explain experiments in which information stored in long-term memory was re-trieved for processing in working memory, including the phenomenon of chunking, whereby long sequences of infor-mation are grouped into shorter chunks [2]. The extended model is presented in Fig. 1.
Of these various components, the phonological loop is the only one directly related to the storage and retrieval of auditory information. While originally intended to model speech processing and retention, Baddeley has suggested that the basic concepts at play may underscore a listener’s com-prehension of music [3] and, to that end, the applicability of the model in musical domains has received intensifying investigation [4,5].
To the extent to which the phonological loop is
inher-ently recursive, it has the potential to offer unique analyti-cal insights into live coding performance practice, in which generative loops and recursive and iterative functions are of fundamental structural importance.
The PhonoLogiCAL LooP
AnD iTS effeCT on WoRKing MeMoRy
Baddeley and Hitch’s phonological loop consists of two stages: a passive short-term storage that acts as a temporary storage buffer for incoming speech and an active articulatory subvocal rehearsal and recoding mechanism in which stored informa-tion is internally repeated until it is phonologically encoded. The relationship between the two is illustrated in Fig. 2.
The phonological loop helps account for a number of work-ing memory phenomena that have received widespread ex-perimental investigation. It has been shown, for example, that the inherent difficulty in accurately remembering words of considerable length [6], known as the word length effect, is explained by the limited temporal capacity of a phono-logical short-term storage facility optimized to store sounds typically not exceeding a duration of 1.5–2.5 seconds [7]. Larsen, Baddeley and Andrade have shown how
phonologi-cal similarity, whereby similar-sounding words or particular
sequences of phonemes are more difficult to remember than those of great contrast, is accounted for by the subvocal re-hearsal mechanism of the model [8]. Interruptions to this rehearsal process also help to explain the impact of
articu-latory suppression and other suppression effects on the
ac-curate retrieval of information [9,10]. Two other important phenomena explained by the model include those related to presentation modality [11], whereby the visual reinforcement of auditory information assists in the accurate retention of information, a phenomenon usually associated with Paivio’s “dual-coding” theory [12], and that of chaining [13], in which the associative relationships between sequential auditory stimuli are established during the rehearsal stage through the attribution of different weightings, a critical process for the accurate recollection of the order of phonemes within a word or numbers within a sequence, for example.
David Kim-Boyle, Sydney Conservatorium of Music, University of Sydney, 1 Conservatorium Road, Sydney, NSW 2000, Australia.
Email: <[email protected]>.
See <www.mitpressjournals.org/toc/lmj/-/27> for supplemental files associated with this issue.
This paper explores how various phenomena of working memory are actively drawn on and can provide useful insights into live coding performance practice. The author argues that the phonological coding processes that cognitive psychologists believe underscore how auditory information is retained and recollected in working memory can enrich our understanding of live coding performance practice, where loop-based processes often provide key structural units. The author suggests that such practice finds structural coherence and aesthetic value through a real-time play with the effects of working memory.
While phenomena such as articulatory suppression, word length effects, phonological similarity, chaining and modality effects are explained by a phonological loop model of work-ing memory, it is important to remember that the phono-logical loop and the vast majority of experiments designed to investigate its explanatory robustness have focused on phonological encoding and various phenomena associated with speech retention. As previously noted [14], while there is a growing body of research investigating the usefulness of the phonological loop in helping to account for musical memory, there has been considerably less investigation into the usefulness of the model for helping to develop insights into musical forms and performance practices such as live coding, in which loops and other iterative functions are im-portant structural determinants. Given the computationally based metaphor of cognitive models of memory more gen-erally [15], it is also somewhat germane perhaps to consider the extent to which phenomena associated with a loop-based memory model can become active performance techniques. Live CoDing AnD The PLAy of MeMoRy
While live coding encompasses a wide range of performance practices [16], programming techniques and languages [17], generative algorithms, modified during a live performance
[18], are typically of central importance. This importance is underlined by the tendency in many live performances to project the code for the audience to view. Given the impor-tant role of loops in much live coding practice, to what extent might current practitioners actively draw on some of the phe-nomena of working memory associated with a phonological loop cognitive model in helping structure performances, and more interestingly, perhaps, to what extent might the inhibi-tion of memory through exploitainhibi-tion of these techniques be directed toward aesthetic ends?
Experiments designed to test the ability to memorize a musical phrase have shown that memorization is facilitated or inhibited by the number of notes the phrase contains and the complexity of the associative relations between those notes. Pembrook has shown, for example, that phrases lon-ger than eleven notes can rarely be remembered accurately [19], while numerous experiments have shown that a random distribution of pitches, for example, are typically much more difficult to memorize than a series of pitches that have or-dered, associative relationships such as those defined by tonal progressions [20]. The length of a musical phrase that can be accurately retained by the listener is a direct manifesta-tion of the word length effect, while associative links between pitches directly recalls principles of chaining. While musical loops can theoretically be of any length, those created by gen-erative algorithms can be more accurately retained by an at-tentive listener if they contain fewer than 11 unique events. In the generative algorithm for Nick Collins’s Algoravethm 11311, programmed in the popular SuperCollider platform [21], for example, arrays that contain data for pitch sequences, rhyth-mic patterns and various other musical parameters typically do not exceed eight items (see Fig. 3).
While prescribing a low limit on the amount of data con-tained in an array facilitates the listener’s retention of that information, accurate retention is also supported through the phenomenon of phonological similarity. While distinctions in pitch within a looped sequence of sonic events are perhaps the most direct manifestation of phonological similarity, distinctions can be made through any number of means. In Klipp AV’s (Nick Collins and Frederik Olofsson) ICMC2007 live coding performance for example [22], timbral differen-tiation between sonic events is far more pronounced, as non-contiguous samples are recursively recalled from an audio buffer, producing fractured sonic textures with constantly shifting timbre. Conversely, looped processes may be dis-guised and memory inhibited by maximizing phonological
fig. 1. Baddeley’s extended model of working memory. (© David Kim-Boyle)
fig. 2. A schematic overview of the encoding process within the phono-logical loop [35]. (© David Kim-Boyle)
similarity as in Thor Magnusson’s Threnoscope [23], which will be examined later.
The Australian live coding performance duo aa-cell (An-drew Brown and An(An-drew Sorensen) actively explore princi-ples of chaining in many of their performances. This usage can range from the use of Markov chains and other probabilistic distribution functions, to weight pitch selections in diatoni-cally organized scale degree arrays, through to the use of lin-ear functions to organize temporal structure [24]. Associative relationships need not necessarily be contained within one sonic property, such as pitch ordering for example, but may be established across auditory domains. In one example by
aa-cell [25], control data used to control the timbral evolution of synthetic drones is also mapped to rhythmic patterns that gradually evolve during performance. Baddeley and Hitch have argued that the phonological loop retains associative relationships through a connectionist-type series of weight-ings ascribed to the sequence of events reinforced during the rehearsal process [26,27]. The manipulation of these weight-ings, such as occurs during a live coding performance, ar-guably forms the primary performance technique employed to provide structural coherence in aa-cell’s performances.
Just as the word length effect, phonological similarity and chaining may provide organizing principles for live coding
fig. 3. SuperCollider code excerpt from Nick Collins’s Algoravethm 11311 [36]. (© Nick Collins)
fig. 4. The graphic interface for Thor Magnusson’s Threnoscope [37], in which different colored bands represent the strength and duration of harmonic components of a drone. Parameters associated with these components are adjusted during performance through a script interface (right). (© Thor Magnusson)
fig. 5. Script excerpt from Sorensen’s Strange Places [38], in Sorensen’s Impromptu [39] programming language. (© Andrew Sorensen) performance practice, modality effects also help reinforce
accurate retention through the multisensory presentation of information [28]. At the same time, modality effects offer a means of underscoring musical coherency. Modality effects are most commonly drawn on through the visual projection of live code and the transparency through which that code and its manipulation may be understood by an audience or observer [29]. Magnusson’s Threnoscope, for example, offers a live coding system in which a circular graphic interface provides a direct visual correlation to scripted code entered in a parallel window; see Fig. 4. The temporal evolution of the sound spectrum is mapped to movement around the circle, with different colored bands providing a direct correlation to harmonic complexity.
Magnusson’s Threnoscope is unusual among live coding systems in that it affords central structural importance to the manipulation of timbre rather than discrete pitched events. Moreover, as a drone-based system, it adopts a particular approach to temporal structure in which discrete rhythmic units are not afforded structural valence. While the loop-based nature of the system aligns it with the phonological loop’s structural model, it is the phonological similarity ef-fect that provides the basis for its afef-fect. Despite its cyclical temporal structure, cycles of the looped drone are made more difficult to perceive through a timbral structure that blurs discrete start and finish times of any looped material. This is particularly intensified through a timbral continuity that inhibits the perception of discrete sonic units.
To the extent to which the visual presentation of code reinforces for the listener the development of a live coding performance, any cross-modal suppression effects, in which retention is inhibited by cross-modal interruption to the subvocal rehearsal process, would seem to be reduced [30]. In most live coding performances, the cross-modal relation-ship of the visual to the sonic is generally intended to act as
a support, a concrete memory or perceptual tracing [31] of an ever-evolving musical process rather than an inhibition.
While the visual artifact, whether in script form or graphic, explicitly provides the opportunity for an audience to better understand how a live coding performance develops, it also serves as a critical reminder for the coders or performers themselves of the musical structure they have developed and are in the process of developing. This necessarily entails that the code be clearly organized according to established con-ventions, efficient and intuitive in order to facilitate real-time editing [32] (see Fig. 5).
ConCLuSion
Phenomena such as word length effects, phonological simi-larity, modality effects, articulatory suppression and chain-ing strongly suggest a phonological loop model of workchain-ing memory. Given the strong evidential support for such a model, the potential for the musical application of these effects to provide unique ways of developing structure in performance practices such as live coding, in which loops are of primary structural importance, is of particular inter-est. In this respect, the play between musical coherency and the constraints of memory forms an aesthetic locus for this practice. Given the cognitive model underscoring many pro-gramming languages [33], live coding can be viewed as the embodiment of a coupled process [34], in which code becomes an active externalism of an internal cognitive process. Con-versely, it is also worth noting the degree to which the type of code and coding processes employed predisposes live coding toward particular types of musical structure. Ultimately, the relationship between code, memory, cognitive processes and musical intention becomes a symbiotic one, where the play of and the active play with memory create unique musical forms with rich potential for further development.
References and Notes
1 A.D. Baddeley and G. Hitch, “Working Memory,” in Gordon Bower, ed., The Psychology of Learning and Motivation: Advances in Research and Theory, Vol. 8 (New York: Academic Press, 1974) pp. 47–89. 2 A.D. Baddeley, “The Episodic Buffer: A New Component of Working
Memory?” Trends in Cognitive Sciences 4, No. 11, 417–423 (2000). 3 A.D. Baddeley, “Alan Baddeley: The Musical Phonological Loop”
(5 November 2010): <https://youtu.be/jzxbFFuIoCs> (accessed Sep-tember 2016).
4 L.M. Thompson and M.J. Yankeelov, “Music and the Phonological Loop,” Proceedings of the 12th International Conference on Music Per-ception and Cognition and the 8th Triennial Conference of the Euro-pean Society for the Cognitive Sciences of Music (Thessaloniki, 2012) pp. 980–986.
5 W.L. Berz, “Working Memory in Music: A Theoretical Model,” Music Perception: An Interdisciplinary Journal 12, No. 3, 353–364 (1995). 6 Note that accurate retrieval of words with many phonemes is affected
more by a word’s temporal duration than by its phonemic complex-ity.
7 A.D. Baddeley and G. Hitch, “Towards a Network Model of the Articulatory Loop,” Journal of Memory and Language 31 (1992) pp. 429–460.
8 J.D. Larsen, A. Baddeley and J. Andrade, “Phonological Similarity and the Irrelevant Speech Effect: Implications for Models of Short-Term Verbal Memory,” Memory 8, No. 3, 145–157 (2000).
9 Z.A. Schendel and C. Palmer, “Suppression Effects on Musical and Verbal Memory,” Memory & Cognition 35, No. 4, 640–650 (2007). 10 P. Salame and A. Baddeley, “Disruption of Short-Term Memory
by Unattended Speech: Implications for the Structure of Working Memory,” Journal of Verbal Learning and Verbal Behavior 21 (1982) pp. 150–164.
11 Salame and Baddeley [10].
12 See A. Paivio, Imagery and Verbal Processes (New York: Holt, Rine-hart & Winston, 1971); A. Paivio, Mental Representations (Oxford: Oxford Univ. Press, 1986).
13 Baddeley and Hitch [7].
14 Thompson and Yankeelov [4]; Berz [5].
15 T. Ormerod, “Human Cognition and Programming,” in J.-M. Hoc et al., eds., Psychology of Programming 307 (1990) pp. 63–82. 16 C. Nilson, “Live Coding Practice,” Proceedings of the NIME 2007
Conference (New York, 2007) pp. 112–117.
17 Popular languages include ChucK, EMacs, SonicPi and Supercol-lider, but other languages such as ixi and Impromptu have been developed by live coding practitioners themselves.
18 A.R. Brown and A. Sorenson, “Interacting with Generative Music through Live Coding,” Contemporary Music Review 28, No. 1, 17–29 (2009).
19 R.G. Pembrook, “The Interference of the Transcription Process and Other Selected Variables on Memory During Melodic Dictation,” Journal of Research in Music Education 34, No. 4, 238–261 (1986). 20 L. Dewitt and R. Crowder, “Recognition of Novel Melodies After
Brief Delays,” Music Perception 3 (1986) pp. 256–274.
21 N. Collins and A. McLean, “Algorave: Live Performance of Algo-rithmic Electronic Dance Music,” Proceedings of the International Conference on New Interfaces for Musical Expression (London 2014) pp. 355–358.
22 Klipp AV, “ICMC2007—Klipp AV,” 2007: <www.youtu.be/zbXCo7 w2Nks> (accessed October 2016).
23 T. Magnusson, “The Threnoscope: A Musical Work for Live Coding Performance,” First International Workshop on Live Programming (San Francisco, 2013).
24 A. Sorenson and A. Brown, “aa-cell in Practice: An Approach to Mu-sical Live Coding,” Proceedings of the 2007 International Computer Music Conference (Copenhagen, 2007) pp. 292–299.
25 aa-cell, “aa-cell live coding at The Loft,” May 13, 2007: <https://youtu .be/OBt4PLUv2q0> (accessed October 2016).
26 Baddeley and Hitch [7].
27 N. Burgess and G.J. Hitch, “Memory for Serial Order: A Network Model of the Phonological Loop and its Timing,” Psychological Re-view 106, No. 3, 551–581 (1999).
28 Paivio [12]; M. Quak, R.E. London and D. Talsma, “A Multisensory Perspective of Working Memory,” Frontiers in Human Neuroscience 9, No. 197, 1–11 (2015).
29 A. McLean et al., “Visualisation of Live Code,” EVA’10 Proceedings of the 2010 International Conference on Electronic Visualisation and the Arts, London, 2010) pp. 26–30.
30 Z.A. Schendel and C. Palmer, “Suppression Effects on Musical and Verbal Memory,” Memory & Cognition 35, No. 4, 640–650 (2007). 31 J.A. Adams, “A Closed-Loop Theory of Motor Learning,” Journal of
Motor Behaviour 3 (1971) pp. 111–149. 32 Brown and Sorenson [24].
33 Ormerod [14].
34 A. Clark and D.J. Chalmers, “The Extended Mind,” Analysis 58, No. 10, 10–23 (1998).
35 Baddeley and Hitch [1].
36 N. Collins, Algoravethm 11311, June 2013. Source code available at <www.composerprogrammer.com/music.html> (accessed Novem-ber 2016).
37 Magnusson [23].
38 A. Sorenson, “Strange Places,” November 2011: <www.youtu.be/TxQ JPNSzl9s> (accessed Oct. 2016).
39 A. Sorenson, “Impromptu: An Interactive Programming Environ-ment for Composition and Performance,” Proceedings of the Aus-tralasian Computer Music Conference 2005 (Brisbane: Australian Computer Music Association, 2005) pp. 149–153.
Manuscript received 2 January 2017.
DaviD Kim-Boyle is an Australian composer and new
media artist. His research and creative practice is primarily concerned with the representation of complex, nonlinear forms with real-time graphic notations. Further information may be found at <www.davidkimboyle.net>.
