chronological overview

Interaction begins with an initial period of playback and preparation, as the composer familiarises himself with the video, aligns patterns to events in the cartoon, and loads samples in the

HALion sampler. This period is characterised by mouse interaction

in the host. Once complete, mouse use is largely limited to repositioning the host song pointer before playback, and otherwise avoided by the composer, who notes that “mouse usage for creative things is a problem" and only accepts its role in the sampler because he doesn’t try to use it creatively.

After this point, the composer spends almost all his time with both hands on the keyboard. Despite the MIDI keyboard beside it, the computer keyboard is used for pitch-entry. Similarly, though his studio contains a control surface, mixer, and many other MIDI synthesizers and keyboards, they remain unused.6

After preparation, just over 15 minutes are spent recreating the

Warner Brothers theme tune for the start of the cartoon. Largely an

exercise in musical transcription, this period is characterised by higher interaction rates and productivity, quickly producing a fully- orchestrated arrangement of the jingle. During this period, the composer does not reference an original recording or score of the music. Instead, the composer enters and edits the music using audio feedback to build a copy of the piece from memory, experimenting with edits and identifying mistakes by ear.

composing by ear _{Consequently, audio feedback is in constant, frequent use, during}

interaction. Playback commands follow even small edits, where it is clear the composer uses the audio to understand the music he has written, relying less on the visual notation. This illustrates the central role of audition in manipulation-driven notation systems (inset, see Figure 4-9). Occasionally, there is more sustained editing between auditions, when the sound, he says, is more predictable.

“expand/explore” approach

This intuitive, exploratory approach to composition is evident throughout the session, as the composer works linearly, drafting and finishing small, sequential sections, rather than creating a blueprint for the whole soundtrack. In subsequent discussions, he observed that the practice of working in small sections is common in tracker users, in contrast to sequencer users, who tend to build pieces in layers (e.g. tracks), commenting:

Actually, unless I'm remixing or rearranging an existing piece, I'm never planning ahead. I don't plan large things. I expand/explore small things.

6

FFOCUS & ACTIVITY Host (Cubase SX)

% time in focus

(music playing) —input (avg. cmds/min)

(measured from top) reViSiT

% pattern editor

(music playing)

% other screens

(music playing) —input (avg. cmds/min)

USER INPUT (cmds/min) Cubase ▬keyboard —mouse reViSiT ▬keyboard —mouse

reViSiT INPUT (cmds/min) Based on contexts of key commands in reViSiT use (see 7.3 and Figure 7-5).

—EDIT —NAVIGATION —PLAYBACK —DATA —SELECTION —CLIPBOARD —FOCUS —SETTING

In reference to vertical and horizontal composition styles (Folkestad, 1996), which respectively correspond to initial focuses on harmony or melody, this approach brings both considerations forward in the composition process. Within the tracker’s pattern architecture, the composer is seen to work horizontally, laying down short excerpts of melody, then augmenting it with harmony and even final touches, before moving to the next pattern. Thus a whole song becomes the product of many smaller, sequential creative processes, where each pattern goes through Sloboda’s progression from an initial draft form to final score (Sloboda, 1985). Moreover, new themes are rarely auditioned away from the pattern (e.g. in the Instrument List), but entered and experimented with in-place, in the pattern itself.

(a)

(b)

(c)

(d)

linear working style _{While constructing patterns could be approached using trial-and-}

improvement, the composer’s ability to maintain a coherent musical thread between patterns, and across the piece, demonstrates a deeper musical understanding. Few mistakes or major corrections are evident; the composer enters the majority of a section in sequence, and seems to have clear idea of what he wants, and how to realise it (an example of Clear Goals, in flow; see Section 3.7). Though the user has had considerable exposure to music performance (including piano tuition), his composition practice is largely self-taught; implicitly learnt over many years of working with trackers ("no training; just looking, listening, seeing and understanding the relation”). This has lead to an extensive, but tacit knowledge of musical processes, which he finds difficult to

Figure 5 – Session profile from video study.

A chronological view of the session, spanning multiple phases of creative editing (see above), presented in five plots (see left for legend):

(a) Focus and overall input activity for reViSiT and Host (Cubase SX), shown as a percentage of the user’s interaction time.

(b) User input in reViSiT and Host, broken down by keyboard and mouse interaction, measured in events per minute.

(c, d, e) reViSiT keyboard interaction broken down by context, based on the descriptive model outlined in Section 7.3 and Figure 7-5, measured in commands per minute.

articulate. For example, asked how he knows patterns will work together, he states, “If the expanding/exploring is done in a natural way, then it'll fit", and says that he simply relies on listening to check that a section feels “natural”.

“macro-listening” _{To gain a broader perspective of the music, the composer devotes}

long periods to repeated playback of the wider song (often more than 30 minutes in length), which he calls “macro-listening”, contrasting to the shorter auditions supporting editing (“micro- listening”). During this time, the task switches from composition to active listening (a realisation-driven system; inset, see Figure 4-9). He also makes extensive use of selections and clipboard, allowing him to work with larger blocks of music and repeat elements of the music, to form progressions. However, users with less experience, lacking such knowledge and technique, may find it harder to maintain themes and ideas across the breaks between patterns. Moreover, this linear workflow may be a consequence of relying on audio, rather than visual, feedback – where the poor role

expressiveness of the text notation makes it harder to step back and

quickly gain a broad overview; with audio, longer perspectives entail longer interruptions, as the song plays in realtime.

“spot-on debugging” _{Instead, the composer uses short excerpts of playback to guide}

edits, and only listens to it in its entirety towards the completion of a phrase. Here, listening becomes the focus of interaction, as he triggers playback (F7) with his left hand and leaves it poised over the adjacent stop key (F8), ready to terminate playback and jump straight into editing, cursoring with his right hand, as soon as a mistake or new idea becomes apparent. The composer calls this technique “spot-on debugging” (in reference to similar approaches in programming, such as just-in-time (JIT) debugging), a further example of the primary role of musical feedback in the tracker. Fast navigation around the music and program is central to the composer’s working style. Rapid, complex cursor activity, seam- lessly interwoven with almost every task (including listening, note entry, arranging, and instrumentation) frequently exceeds rates over 100 cmds/min. In spot-on debugging, for example, cursors are used to quickly select the playback material and then to quickly convey the composer to the appropriate point, when he hears something.

near-realtime

composition During note entry, the cursor is also used to step through the

pattern to correctly place notes. Unlike a live recording, notes are not entered in realtime, but the composer’s dexterity in interleaving cursor movement with note entry allows him to preserve much of the rhythm of the notes, so the character of the melody or phrase is preserved in the incidental audio feedback. At the same time, the

lack of rigid metre allows him to slow down or pause as necessary, for more complex edits. Faster-than-realtime input is also possible, and it is not uncommon to see longer passages initially entered into a confined space, then expanded using shortcut keys.

arrangement and abstraction in clipboard use

In the session profiles (Figure 5), some editing periods are characterised by direct data entry, and others by increased use of selections and the clipboard. Frequently, the composer is seen to edit a short section in detail (a beat or bar) before cloning it to form the basis for longer phrases. Though this practice is common in loop-based music, leading to progressive musical styles7, the composer uses this approach for more intricate musical structures, whereby the flexibility of block selection and the clipboard allow him to build new patterns not just by repeating whole sections, but by drawing on and mixing select parts of previous material, in a process more like bricolage (see Turkle and Papert, 1992). Unlike individual notes, selection-based edits do not automatically trigger audio feedback, so the composer relies more heavily on short excerpts of song playback and “spot-on debugging”.

After the composer has laid down several basic themes, a slight shift towards increased clipboard use occurs (~02:20), continuing until the end of the session, as new material increasingly draws on that preceding it. During selection use, the interaction rate remains high, and with each key command now affecting multiple notes, overall productivity increases. Block selection supports a subtly higher level of music editing that mixes microscopic note-level editing with more abstract editing based on themes, phrases, parts, and other musical devices. This transition is implicit, with little change in interaction style (input mode, visual representation), thus enabling free movement between stages in the creative process. So, even as a user moves from exploratory creativity (finding themes) into a later-stage composition process based more on problem- solving (arrangement, applying music to video), there is little to hinder them from experimenting with new ideas.

host-based song

& video playback Working with video, the composer is forced to return to the host

program to audition the song in-sync with the visual footage. This diversion punctuates longer periods of interaction with the tracker, in which the music is created and edited. As a result, a clear distinction in the role of playback emerges between the host and

reViSiT, whereby the sequencer provides the longer, broader

musical context, managed through the timeline and transport bar, and the tracker provides focused feedback for editing, through the

7

Music based on a progression, where several iterations of a passage are gradually developed or varied, in respect of melody, harmony, rhythm or texture; common in dance, house, trance, drum‘n’bass music.

keyboard. These two modes of playback differ in frequency, duration, and manner of control, as well as the subsequent posture of the user. In the tracker, the composer continues to interact or hovers, poised over the stop key, in anticipation of further editing. In the sequencer, the composer positions the playback cursor with the mouse, and triggers the song with the keyboard – striking the space bar with his left hand, as it retreats from the keyboard – and then remains idle, listening to the music. In this scenario, the sequencer’s role is that of a tool for evaluation, the final stage in the creative process. Later analyses explore this in the context of what other studies (Blackwell and Green, 2000; Smith et al, 2009) have identified as a tendency for music software to focus on the later stages of creativity (i.e. transcription, productivity).

energy and

tiredness At the same time, longer auditions can be restorative. The

composer noted that the rapid interaction and constant focused attention of tracking can be tiring, disposing him towards longer auditions as a productive means of resting. The intense, hard cut bursts of sound arising from frequent auditions of notes, passages and patterns may also lead to ear fatigue, though longer breaks after several hours of interaction help to combat the risk.

centralised

focus & control Within reViSiT, the composer spends the vast majority (93.8%) of

his time in the Pattern Editor. Apart from the initial configuring of instruments and occasional edit to the Pattern Order, the only significant use of any other part of the program is the Instrument List’s role in changing the current instrument used for editing.8 In

reViSiT, there are a number of ways to do this from the Pattern

Editor itself, and while the Instrument List may have its advantages, the composer concedes that his choice of method is likely a habit picked up in IT2, from which the original inspiration for reViSiT’s UI comes. This is a clear indication of well-learnt interaction, based in the development of both motor skills (key sequences) and spatial schemata (the instrument list and keyboard layout).

mastering

the tracker The composer is conscious of his expertise; as something that has

taken years to develop and mature, largely learnt through practice and experimentation, but also through dissecting the music of others’ and the sharing of tips and tricks in online communities (e.g. the demoscene). When asked to reflect on the most important concepts and lessons a new user should learn to develop mastery of the tracker, in comparison to other digital music practices, he cites (in no specific order):

8

More usually, he "picks up" an instrument from existing music in the pattern, by moving to one of its notes and hitting Enter. This way, the visual search through Instrument List is avoided, and the user’s attention can remain with the editing context.

• the freedom and blank canvas of the pattern

to place any note(s) of any instrument in any cell or channel, allowing the composer to group elements as they see fit, without being bound to or separated by MIDI channels, or having to create and prepare tracks before data can be input; • “spot-on debugging”

the rapid edit-audition cycle and use of editing cursors to quickly trigger playback (F7), during which the user listens and remains poised, ready to jump back to editing;

• fast navigation using the keyboard

allowing routes though the program, commands, and sequences of actions to be executed from memory without visual inspection, and fluidly interwoven with other inherently keyboard-based tasks, such as editing.9