axis: The axis represents the number of data dimensions in a dataset The

raw EEG data has a typical frequency range between 1 and 70 Hz and is generally filtered into sub-frequency bands. The six classic clinical EEG bands are as follows: delta, 2-4 Hz; theta, 4-8 Hz; alpha, 8-12 Hz; beta1, 13-21 Hz; beta2, 21-30 Hz; and gamma, 30 to 70 Hz. Besides these six EEG bands, it is not unusual to subdivide these bands further, for example down to 1 Hz bands. Thus for a single channel (i.e. single measurement location) of EEG, this would give between 1 and 40 dimensions per channel.

When there is more than one channel, then three new comparative dimensions can be derived. These are: amplitude asymmetry (the relative power between left and right homologous sites e.g. F3 in comparison to F4); coherence (a measure of the degree of association between two different parameters) and phase (the delay or “lag” between two channels) Therefore with every additional channel there is a multiplication of these three relationships for each frequency band.

So for example, a 4 channel system with 6 EEG bands, would give 6 Absolute Power variables, 6 Relative Power variables and 15 Power Ratio variables

(Power Ratios are a measure of the relationships between different frequency bands in the same location. For example: the theta/beta ratio is computed by dividing the theta power by the beta power). This makes a total of 27 variables for each channel (these can be seen as „Within Channel variables‟)

Since the 4 channels have 6 connections (or links) between each channel and each would have a Coherence value for each of the 6 EEG bands, plus 6 for Phase Difference, making a total of 12 variables for each connection (Between Channels). Therefore a system with just 4 channels would have 216 data dimensions.

Whereas a typical 19 channel EEG system with 6 EEG bands would give 171 links and 3591 data dimensions a 19 channels system with 30 EEG bands would give 15,903 data dimensions and a 256 channels system with 40 EEG bands gives 3,923,712 data dimensions.

This suggests that according to de Campo‟s, ‟Data Sonification Design Space Map„ that EEG sonification lies mostly in the ‟Continuous„ data representation space but overlaps the border with ‟Model-Based„ and ‟Discrete Point„ data representation.

The Z-axis in Figure 3.5.1 represents the number of simultaneous streams suitable for a meaningful data representation. In a visual display for neurofeedback for example, it might be usual to have 3 or more concurrent streams of EEG band power, for example theta, alpha and beta displayed at the same time, and to ask the trainee to focus on, for example, increasing the power of the alpha whilst simultaneously lowering the theta and beta. The sonification of EEG offers the potential to present multiple parallel streams of EEG data and this could

assist in creating a meaningful gestalt out of the complex EEG data. It is an empirical question to see how many simultaneous streams of EEG sonifications people would be able to comprehend and clearly this would be dependent on the type of sonification.

3.5.3.

EEG Sonification Techniques

Over the last 83 year history of EEG sonification, one of the principal motivations cited by authors for proposing the use of sonification to “display” EEG has been to reveal the temporal complexity of the EEG signal. Authors argue that this temporal complexity is lost in visualization techniques and suggest that the human auditory system is particularly well suited to the perception of EEG.

This section will focus on the subcategory of sonification techniques that are capable of the real-time presentation of the EEG. (More information on these techniques is given in Appendix: A2.4 Sonification Techniques).

The 21 real-time EEG sonification techniques found in the literature survey have been categorised for the purposes of this review using de Campo‟s sonification design space map into three broad groups (Discrete-Point, Continuous and Model-Based).

As can be seen in table 3.5.3 below; straightforward audification is the earliest, and one of the most popular, sonification techniques, with six examples occurring in the literature survey; this popularity may be as much to do with its simplicity to implement as its utility.

Only six of the sonification techniques have been used in a neurofeedback study. With the most popular being Amplitude Modulation with five

neurofeedback studies and both Frequency Modulation and threshold sonification technique with three neurofeedback studies each.

Of the 14 EEG sonification neurofeedback studies only 12 will be reviewed in this chapter as the Le Groux, 2009 and Trevisan, 2011 studies did not attempt to validate their work or provide sufficient detail to allow analysis.

Although de Campo‟s sonification design space map helps, from reviewing the EEG sonification literature it would be very difficult to draw a conclusion as to which technique is most appropriate for a particular application. There is no standardised framework for reporting or categorising the different sonification techniques and very little quantitative evaluation of different techniques. Without this necessary foundation it can be difficult to know which sonification techniques to use for a new EEG sonification task.

Table 3.5.3: Shows the 21 sonification techniques that have been used to display real-time EEG. The bold blue text highlights the 14 neurofeedback studies and the numbers in the brackets show total number and neurofeedback studies. The sonification techniques categorised according to de Campo‟s sonification design space map into three sub-groups; Discrete- Point, Continuous and Model-Based. See section 3.5.1 de Campo‟s definitions and appendix “A2.4 sonification techniques” for a description of the different techniques.

Sonification: Study name:

Parameter -Mapping Sonification C o n tin uo us

Audification Adrian 1934, Jovanov 1998, Olivana 2004, Baier 2005, Wu 2009, Khamis 2012,

Amplitude Modulation (7/5) Hardt 1978, Baier 2005, Hinterberger 2011/16, Choi 2011, Hardt 2012, Wang 2013 Frequency Modulation (7/3) Fell 2002, Hinterberger 2004, Wu 2009, Miranda 2010, Trevisan 2011, Hinterberger 2011/16, Lu 2012

Filtered Sonifications (1/1) van Boxtel, 2012 Spectral mapping Hermann 2002 Distance matrix Hermann 2002 Differential Hermann 2002 Neurogranular sample Grant 2000 Timbre mapping Baier 2005

Parameter mapping (2/1) Hermann 2006, Ramirez 2015

D isc re te P o in t

Event-based/Threshold (10/3) Nowlis 1970, Schwartz 1976, Jovanov 1998

Allen 2001, Arslan 2006, Baier 2006, Brouse 2006, Baier 2007, Franco 2015, Chen 2015, Auditory icons Salter 2008

Earcons Jovanov 1999

Flanging (2) Arslan 2005a, Arslan 2005b

Granulation (3) Arslan 2005a, Arslan 2005b, Filatriau 2006 Extrema detection Hinterberger 2004

Model

-

Based

Generative rules music Brooks 2007 Kernel regression Hermann 2008 Tristimulus synthesizer (1/1) Le Groux 2009 Overtone mapping Terasawa 2012 Spatial location Baier 2007

Assessment of EEG Sonification