Processing Naturally-occurring Brain Signals

The main feature shared by the BCI research presented in this section is that all the systems described aim to allow the user to behave naturally. In other words the role of the computer is to

recognise and respond to behaviour-related brain phenomena extracted from a stream of detected EEG data. From a high level, systems designed to process naturally occurring brain signals can be envisaged as human-machine systems where the computer is doing all of the work, adjusting its behaviour to suit that of the user. This is a very different style of interaction than is common with existing methods of human-machine interaction, where the user has to modify his or her behaviour to match the requirements of the machine.

A “thought-induced” computer action may be no more complex than notification within an application that the magnitude of some EEG component’s amplitude has passed above or below some predefined threshold value. Such a switching signal (which is effectively the result of notification of a signal passing a single threshold value) can be used to instigate some action within the application. These systems require provision for setting threshold values for each EEG component signal to be used as switch. The requirement to be able to set reward thresholds is a feature we have already encountered in the goal-driven biofeedback system interfaces described previously.

More complex brain-signal processing systems require the incorporation of neural network technology in order to try to recognise a user’s particular thought or action. This second approach to utilising naturally occurring brain activity, though more ambitious, is nevertheless yielding some interesting early results. However, we will begin by looking at systems created to investigate simple EEG component amplitude-based switching.

It is widely accepted that the earliest investigation into the use of EEG component amplitude to control an electronic device was carried out by Edmond Dewan (Dewan 1967). Dewan filtered alpha activity from the EEG, connecting the output from the filter to a Schmitt trigger, which generated an electrical pulse whenever the amplitude of the alpha signal exceeded a predefined threshold value. The pulse was fed to a computer, which was programmed to translate pulses into Morse code dots or dashes (depending on the pulse duration).

During evaluation of the system each subject’s task was to transmit, in order, the letters of the alphabet using only alpha-generated Morse code. Each subject was instructed to look upwards with eyes closed to achieve maximum signal voltage (alpha present = pulse on). For minimum voltage (alpha absent = pulse off) subjects were either to open their eyes and fixate them on a nearby object, or to fixate closed eyes on an imaginary point some distance in front of them.

The results of Dewan’s study where not overly promising in that 33% of all Morse characters transmitted to the computer were erroneous and had to be re-sent. Taking into account the time to re-send the correct character, the inter-character pause period and time allowed for noise present in the signal, it was found to take approximately 35 seconds to transmit a single correct letter using alpha signals only. Of course the bottleneck in a system designed to respond to alpha amplitude switching is the binary nature of the control signal. A two-state system is limited by the speed at which a user can close and open his or her eyes and does not therefore provide the best mechanism for transmitting text. This has not, however, deterred others from exploring brain- computer interaction based on alpha signal switching (Kirkup 1997, Patmore 1994).

In 1988 Farewell et al. (Farewell 1988) developed a system to allow disabled users to input Romancharacters into a computer using a single, recognisable evoked brain potential. Farewell’s system used an event-related brain signal known as the P300 response (so called because it appears as a spike in the EEG signal 300 milliseconds (mS) after a stimulating event). The interface to Farewell’s system consisted of a 6-by-6 matrix presented on a computer screen, which contained the 26 letters of the alphabet (see Figure 3.4). In order to select a particular letter, a subject had to focus their attention on that letter for the duration of a selection period. During that selection period each row was intensified for a period of 100mS in turn. When all of

to ascertain whether a response was present for a given row or column. The final inter-stimulus period (ISI) used was 500mS.

MESSAGE

BRAIN

Choose one letter or command

A G M S Y * B H N T Z * C I O U * TALK D J P V FLN SPAC E K Q W * BKSP F L R X SPL QUIT Figure 3.4 Farewell's soft keyboard

EEG data was detected from an electrode positioned over the occipital lobe16. Pre-processing involved filtering and amplifying the signal after which real-time analysis of signal data could be carried out. The system used a covariance algorithm to identify which letter had elicited a P300 response. Using this system, four able-bodied subjects successfully completed the task of spelling the work BRAIN using only their P300 response. However, the character identification (and thus

transmission) rate of the system was little better that Dewan’s, at 26 seconds per character.

Four years later, researchers at the Smith-Kettlewell Eye Research Institute (Sutter 1992) attempted to improve on Farewell’s character transmission rates. The interface to their system consisted of an 8-by-8 matrix containing labelled fields or keys. Beneath the initial matrix were constructed 32 levels of key functionality. So, for example, the 1st _{key overlay contained the}

alphabet and a set of most frequently used words. If the word that the user required was not on the first screen, then he or she selected the first letter of that word. The system then switched to an overlay of words beginning with that letter, and so on. In total, the system provided access to 700 commonly used words.

16 _{see Appendix B}

A prototype system was evaluated using 20 disabled and 70 able-bodied subjects. After a training period of up to one hour, the system could recognise which cell in the matrix an able-bodied subject was focussing on in between 1 and 3 seconds. Muscle generated signal contamination proved to be a problem with disabled subjects, and this was overcome in one instance by detecting EEG using an inter-cranial implant. This implant remained in place for an 11-month evaluation period, toward the end of which the subject achieved communication rates of around 11 words per minute using the system.

Keirn and Aunon (Keirn 1990) chose a different type of brain response to study in order to identify patterns of brain activity associated with particular mental tasks. Their notion was to translate distinct brain responses into a computer control language. The underlying principle behind this system was one of hemispheric specialization – effectively the ability to distinguish between a task which involves activity mainly in the left hemisphere of the brain, and a task that involves activity mainly in the right hemisphere (Springer 1993). Keirn’s notion was that if it were possible to distinguish between a left brain task, a right brain task and a third resting or no- activity state, then it would be possible to construct a language of commands such as:

Let the letter A signify the detection of a right hemisphere task Let the letter B signify the detection of a left hemisphere task

Let the letter C signify the detection of a resting condition for x seconds

Which could be translated into a command sequence such as:

AC = Stop CAB = Go

BAC = turn right 90 degrees etc…

Subjects were given five mental tasks to perform. The first was a baseline measure of activity and involved the subject relaxing and trying to think of nothing at all. The second task was a non- trivial multiplication problem to be solved mentally. For the third task, a drawing of a three- dimensional object was given to the subject to study for 30 seconds, after which he or she was instructed to visualize the object being rotated about an axis. Mental composition of a letter to a

In all of the tasks the subjects were instructed not to vocalize or move. EEG information was detected from a number of electrodes positioned across the surface of the scalp. Further electrodes detected signals generated by muscle and eye movement and these signals were eliminated, in real-time, from the EEG data.

A Bayes Quadratic Classifier (BQC) was used to test task accuracy against feature sets created from estimates of the spectral density of EEG in four frequency bands – delta, theta, alpha and low/middle beta. Keirn and Aunon were surprised to find that after a training period, the classifier could distinguish between all five tasks, with an accuracy of over 95% based on only 2 seconds of data. Unfortunately, Keirn and Aunon appear to not have pursued their studies beyond this point.

In 1994 Pfurtscheller et al (Pfurtscheller 1993, Pfurtscheller 1994) used mu-rhythm blocking in the development of a system known as the GRAZ Brain Computer Interface. Mu is a distinctly pattered movement-related brain signal appearing within the alpha frequency band17. Khulman (Kuhlman 1978) had previously established that contralateral18 _{mu-rhythm blocking (i.e. notable}

drops in mu signal amplitude) occurred during the 1-second period prior to motor activity. Pfurtscheller’s group set out to establish whether it was possible to distinguish between three distinct physical actions by studying the brain’s mu rhythm responses alone.

For the study, 30 electrodes were applied in a rectangular array over the left and right post-central (roughly beneath the top part of the skull) areas of each subject. Features related to mu-rhythm blocking were extracted from the EEG data stream and passed to a pattern classifier. An initial training period for each subject involved using data from all 30 electrodes to train a Learning Vector Quantifier (LVQ) artificial neural network.

During a trial, subjects were visually cued to perform one of three movements.

1. Press a micro-switch with the right index finger 2. Press a micro-switch with the left index finger 3. Flex the toes of the right foot

17 _{Alpha is detected over the visual cortex of the brain, whereas mu activity is detected over the motor cortex.}

18 _{Occurring in the brain hemisphere opposite the limb moved i.e. left hand produced mu-blocking in the right} hemisphere of the brain

After the visual cue appeared, a period of 125mSec followed during which the subject was to mentally prepare for movement. After this period, the cue disappeared from the screen and the subject was expected to start the movement. As is the case with pattern classification based on neural networks, each movement had to be repeated by a subject at least 30 times before the system was able to recognise it.

Khulman’s findings were confirmed – that the final step in this cycle (the actual movement) is not necessary for the invocation of mu-rhythm blocking. It is the planning of movement that elicits

the response. Discrimination between right and left finger movement using information within the 10-12 Hz frequency band alone was possible to an accuracy of 83%. Discrimination between all three different movements revealed a less impressive classification accuracy of between 51 and 58% using the alpha component alone. However, when the classifier was reconfigured so that it was looking at data from alpha, beta and gamma bands together, the accuracy rate improved to between 62 and 70%.

Pfurtscheller’s study has recently been repeated by researchers at Imperial College who have created their own neural-network-based system for processing movement-related brain responses (Penny 1996a, Penny 1996b, Penny 1998). Their first system relied on processing data from a 22- electrode array. As with Pfurtscheller’s study, subjects were to repeat a particular task (movement of the right hand for instance) at cued intervals. The aim was to train the network to recognise from the EEG pattern of activity when movement of the right hand was occurring. The second task subjects were instructed to undertake was to imagine the same movement, again on cue. The

results from this study again confirmed that these imagined movements produce exactly the same patterns of EEG activity as the actual movements.

Due to the amount of data processing that must take place in order to decipher the information from the EEG detected from 22 positions on the scalp, systems such as these have to process their data off-line. A second simpler system was subsequently developed by Imperial, which relied on only 3 electrodes and carried out real-time, on-line signal processing on the detected EEG data. This data was used to demonstrate 1-dimensional cursor control on a computer monitor. The activation mechanism required the subject to use mental imagery to produce two discernible EEG

arithmetic. Recognition of one task was translated into upward vertical motion of a cursor, recognition of the other into downward motion.