Automatic error recovery using P3 response verification for a brain-computer interface

(1)

RIT Scholar Works

Theses

Thesis/Dissertation Collections

2004

Automatic error recovery using P3 response

verification for a brain-computer interface

Samuel A. Inverso

Follow this and additional works at:

http://scholarworks.rit.edu/theses

This Thesis is brought to you for free and open access by the Thesis/Dissertation Collections at RIT Scholar Works. It has been accepted for inclusion

in Theses by an authorized administrator of RIT Scholar Works. For more information, please contact

[email protected]

.

Recommended Citation

(2)

Automatic Error Recovery

Using P3 Response

Verification for a

Brain-Computer Interface

Rochester Institute of Technology

Computer Science Department

Master of Science Thesis

Samuel A

.

Inverso

May 9

,

2004

In

p

ll

rtial

f

u

lfillm

e

nt

of th

e

r

equir

e

m

e

nts for

t

h

e

d

egr

ee

of

Master

of

Scien

ce

.

Michael Van Wie

L2-/

M

C.

7

tor

R

ea

der:

Micha

e

l

V

a

n Wie

,

PllD

Roger S. Gaborski

(3)

I,

Samuel A.

Inverso, do hereby grant permission to

copy this

document, in

who

l

e or

in

part

,

for a

ny

non-

c

ommercial or non-profit purpose. Any other use

of this document requires the written permission

of the

author.

Samuel A. Inverso

(4)

First,

I would like _{to thank my advisor,} Jessica

Bayliss,

for the support, free

dom,

andguidance_{necessary to} succeedin this

interdisciplinary

field. Jessica's

straightforward advice and _easy communication style created an environment

where the risk inherent in

interesting

research was _positivelybalanced against

theshort timeframe imposed

by

aMaster's degree.

I would like to thank Gerwin Schalk of the Wadsworth Center for

help

in understanding BCI2000'ssubtleties and complexities, and hisoften immediate

responses to email queries. In addition, I would like to thank Agustin Roca

and

Vijay

Kenkat ofTucker-Davis Technologies who were _{exemplary in their} customer service

diagnosing

and _{anticipating hardware}and softwaredifficul tieswiththe amplifier,and providingindispensableaidin thesignal acquisition system'sdesign.

Iwouldlike to thankCarol Marchetti for herelucidationof statistical signif

icance_tests, which was_{instrumental in understanding}and_{applying the}correct

statistical methods in dataanalysis.

I would liketo thank RebeccaAllen for her allowances and support

during

myabsencefromthe Media Lab Europe while

finishing

this thesis.

I would like to thank Andrea

Chew,

Chad

Hulbert,

and Juanjo Andres Prado forproof_{reading this} material. Special thanks to Chad for mercilessly

scrutinizing this

document,

which leadtocorrections and suggestionsthathave greatlybenefited its presentation.

Thanks to _{Dan Kunkle for challenging} metoachieve and excel throughout this degree.

Iwould alsoliketothankalltheprofessors,students,

family,

andfriendswho accompanied me throughout this work.

They

supported me academically, un

derstoodmyreclusive

behavior,

andoffered littlewordsof encouragementwhen motivation was minimal. Though theirnamesarenot

listed,

eachimpactedthis thesis in ameaningfulway not _easilyexpressed inwords.

Finally, I would like to_{thank my} parents, Ronald and

Connie,

whose_early

and consistent support ofeducation instilled in me afortitude without which

(5)

A brain-computerinterface

(BCI)

isan augmentative communication mech anism that does not _rely on peripheral nerves or muscles. Current BCIs are error prone and slow with error rates of 10 to 30% and transmission rates of

10-25

bits/min,

however,

error_recovery and correction inBCI has

largely

been

neglected. Thefocusofthis thesis isthedevelopment of amethodtoautomat

ically

recover errorsin BCIusing the P3 brainsignalfor response verification.

TheexistenceoftheP3 signalinresponsestocontrolledgoalitems isshown

in an evoked potential BCI used to control items in a virtual apartment. A

reduced response exists whenitems are accidentally controlled. Offline experi ments were_run,andwithatheoreticalmeanimprovement_{in accuracy}from 78% to

85%,

therewasa_{statistically}significantimprovement (P<

0.008,

Wilcoxon

signed rank _test) _{in accuracy} of 3% _using a correlation algorithm _{for P3 sig} nal detectionon responses. The presence ofthe P3 signal in responses togoal

items indicates it can be used for automatic error _recovery without _requiring additional _time, which will improve thespeed and _accuracyofbrain-computer

(6)

Acknowledgments iii

Abstract iv

List ofFigures vii

List ofTables xiii

List ofAcronyms xvii

1 Introduction 1

2 Brain-Computer Interfaces

(BCIs)

5

2.1 The Physical Brain 6

2.2

Measuring

Brain

Activity

. 8

2.3

Recording

EEG

Activity

. 12

2.3.1 Electrodes 13

2.3.2 Electrode placement . 13

2.3.3 Referenceand Bipolar Recordings 14

2.3.4

Grounding

15

2.3.5 Artifacts . 15

2.4 Electroencephalogram

(EEG)

16

2.4.1 Rhythmic Brain

Activity

16

2.4.2 Event Related Potential

(ERP)

. 19

2.4.3 Event-RelatedDesynchronization

(ERD)

andEvent-Related

Synchronization

(ERS)

21

2.4.4 Other BCIapproaches _usingEEG 22

2.5 Errors in aBCI 23

2.6 BCI Performance Measures . 23

3 Related Work 25

3.1 Choice Organization 26

3.2 Manual Error

Recovery

27

3.2.1 Undo 28

(7)

3.3.1 Error-Related

Negativity

(ERN)

3.3.2 Error Potential 30

3.4 Conclusion 31

4 Automatic Error

Recovery

Using

the P3 signal 33

4.1 Introduction . .... 33

4.2 TheVRapartmentexperiment . . .35

4.2.1 Experimental

Setup

35

4.2.2 TheExperiment . 37

4.2.3 TheAppearanceoftheEvoked Potential P3 Component 38

4.3 Methods . 39

4.3.1

Using

anEvoked-Potential for Error

Recovery

39

4.3.2 Offline Analysis 39

4.4 Results . . 40

4.5 Discussion and Future Work 42

5 Chess BCI Platform for Experimentation 45

5.1 Patient's Brain-Computer Interface 46

5.2 IMPBN's Chess BCI Requirements 46

5.3 Chess BCI Design . 46

5.4 Implementation 48

5.4.1

Proxy

Engine . . 48

5.4.2 Winboard Modifications . . .48

5.4.3 BCI2000 Modules Implemented . . 49

5.5 Discussionand Future Work . . . 49

6 Conclusion ₅₁

6.1 Future work ₅₂

A

Accuracy

Equations ₅₃

B Data used for Response Verification Analysis of the Virtual

Apartment Experiment Screen Condition 57

B.l Peak Pick B.2 Correlation

58 68

B.3 Correlation Indeterminate . _7g

(8)

2.1 General BCI frameworkproposed

by

MasonandBirch[40]. Shows theflowofinformation fromthebrainto thedeviceandfeedback

fromthe deviceto theperson. 6

2.2 BrainStructures: cerebrum, cerebellum,limbicsystem,andbrain

stem. Used with permission[3]. 7

2.3 Projection neuron_{showing information flow. Used} with permis

sion [27]. 8

2.4 The brain's functional lobes. Usedwith permission [27]. 9

2.5 Example brainscans

depicting

spatial resolution. Computer To

mography

(CT)

isan

X-ray imaging

technique thatdoesnot pro vide functionalinformation. Usedwith permission. 10 2.6 Spatial

(mm)

vsTemporal

(sec)

Resolutions for Brain

Graphing

Methods. Usedwithpermission [27]. . . 11

2.7 International 10-20 SystemofElectrode Placement,

(a)

Top

view

showing the 21 electrode positions,

(b)

Side view _showing elec

trodepositionsat10and20%ofthe distancebetweenthenasion andinionmeasured overthe _topofthe head. Usedwith permis

sion [27], 14

2.8

Delta,

Theta,

Alpha,

and Beta rhythmic EEG

frequency

ranges.

Usedwith permission [27]. 17

2.9 P300waveform. 19

2.10 FarewellandDonchin'sP300spellerinterface. Rowsand columns

intensify

ever 125ms. Imageshows rowintensification [24]. 21

3.1 Example chess _opening moves. Screen capture from Vektor3

3.1.6. . 27

3.2 Examplecontrolflow from Wolpawet al.'s manual responseveri ficationina2-Dcursor controlBCI [82]. Theuser wantstoselect A. Iftheusermovesthe cursorto Ain theupperleft corner, the A isselected andthe useris thenpresented withtheresponseto

(9)

sign, sign,

lamp

this scene,

red sphere onthetelevisionset is blinking. 36

4.2 a) Thegrand averagesforcontrol of goals at sitePz (solid

lines)

shown

withthegrand averagesfornon-goals(dashed

lines)

ineach experimen

talcondition,

b)

Thegrand averages over nine subjectsforresponses

togoalitemcontrol(solid

lines)

and mistakesin itemcontrol (dashed

lines). 38

4.3 Responseverification_accuracyvsoriginal_accuracy_usingthecor relation method. Solid lineshows theoriginal_accuracy vsitself.

Itemsabovethe line increased inperformance. . . 41

5.1 Chess Brain-Computer Interface High Level Design . 47

B.l Accuracies _{resulting from varying} average peak for Peak Pick

algorithm. Regressed_true, EOG Threshold: 90 59 B.2 Response Verification

Accuracy

versus Original

Accuracy

for P3

Found and P3 Absent methods (site Fz). Condition:

Monitor,

Algorithm: Peak

Pick,

Regressed: _yes,EOG Threshold:

90,

Peak

Threshold: 61. _q\

B.3 _{Response Verification}

Accuracy

_versus _Original

Accuracy

for P3 Found and P3 Absent methods (site Cz). Condition:

Monitor,

Algorithm: Peak

Pick,

Regressed: yes,EOG Threshold:

90,

Peak

Threshold: 61. _gj

B.4 Response Verification

Accuracy

versusOriginal

Accuracy

for P3 Found and P3 Absent methods (site Pz). Condition:

Monitor,

Algorithm: Peak

Pick,

90,

Peak Threshold: 61.

Accuracy

versus Original

Accuracy

for P3 Found andP3 Absent methods (site Any). Condition:

Monitor,

Algorithm: Peak

Pick,

Regressed: yes, EOGThreshold: 90 Peak Threshold: 61.

B.6 AcrossAll _{Subjects Response Verification Trials} _Goal

and Non-GoalGrandAverages.

Keep

responseif P3wasfoundat siteAny. Condition:

Monitor,

_Algorithm: _Peak

Pick,

Regressed-yes EOG Threshold:

90,

PeakThreshold: 61

B.7 _{Response Verification}_Trials_Subject₁

Goaland_{Non-Goal Grand} Averages.

Keep

responseifP3wasfoundat siteAny.

Condition-Monitor,

Algorithm: Peak

Pick,

Regressed: _yes,EOG

Threshold-90, Peak Threshold: 61

B.8 ResponseVerificationTrialsSubject2Goaland_{Non-Goal Grand} Averages.

Keep

responseifP3wasfoundat siteAny.

Condition-onTT

^g0ruh^: ?eak

PlCk'

ReSressed:

yes,EOG 90, PeakThreshold: 61

62

63

(10)

Averages.

Keep

responseifP3was foundatsiteAny. Condition:

Monitor,

Algorithm: Peak

Pick,

Regressed: yes,EOGThreshold:

90,

Peak Threshold: 61 64

B.10 Response Verification Trials Subject 4 GoalandNon-GoalGrand

Averages.

Keep

responseifP3wasfoundat siteAny. Condition:

Monitor,

Algorithm: Peak

Pick,

Regressed: _yes,EOGThreshold:

90,

B.ll Response Verification Trials Subject 5 GoalandNon-Goal Grand Averages.

Keep

responseifP3was foundat siteAny. Condition:

Monitor,

Algorithm: Peak

Pick,

90,

B.12 Response Verification Trials Subject 6 GoalandNon-GoalGrand Averages.

Keep

responseifP3wasfoundatsiteAny. Condition:

Monitor,

Algorithm: Peak

Pick,

90,

B.13 Response Verification Trials Subject7 GoalandNon-GoalGrand Averages.

Keep

Monitor,

Algorithm: Peak

Pick,

90,

B.14 Response Verification TrialsSubject 8 GoalandNon-GoalGrand Averages.

Keep

Monitor,

Algorithm: Peak

Pick,

Regressed: _yes,EOGThreshold:

90,

B.15 Response Verification Trials Subject 9 GoalandNon-GoalGrand Averages.

Keep

responseifP3wasfoundat site Any. Condition:

Monitor,

Algorithm: Peak

Pick,

90,

PeakThreshold: 61 . 67

B.16 Accuraciesresultingfromvarying thresholdfor Correlationalgo rithm. Regressed true, EOG Threshold: 90 69 B.17 Response Verification

Accuracy

versus Original

Accuracy

for P3

Found and P3 Absent methods (site Fz). Condition:

Monitor,

Algorithm:

Correlation,

Regressed: yes, EOG Threshold:

90,

CorrelationThreshold: 0.68. 71

Accuracy

versus Original

Accuracy

for P3 Found and P3 Absent methods (site Cz). Condition:

Monitor,

Algorithm:

Correlation,

90,

CorrelationThreshold: 0.68. . . 71

B.19 ResponseVerification

Accuracy

versus Original

Accuracy

for P3 Found and P3 Absent methods (site Pz). Condition:

Monitor,

Algorithm:

Correlation,

Regressed: _yes, EOG Threshold: _90,

CorrelationThreshold: 0.68. . 72

B.20 ResponseVerification

Accuracy

versus Original

Accuracy

for P3 Found and P3 Absent methods (site Any). Condition:

Monitor,

Algorithm:

Correlation,

90,

(11)

Goal Grand Averages.

Keep

P3

Fz. Condition: Monitor,Algorithm:

Correlation,

Regressed: yes,

EOG Threshold:

90,

Correlation Threshold: 0.68 . . 73

B.22 ResponseVerificationTrials Subject 1 GoalandNon-Goal Grand Averages.

Keep

responseifP3was foundat site Fz. Condition: Monitor, Algorithm:

Correlation,

Regressed: yes, EOG Thresh

old:

90,

Correlation Threshold: 0.68 73

B.23 Response Verification Trials Subject 2 GoalandNon-Goal Grand Averages.

Keep

response ifP3 was found at site Fz. Condition:

Monitor,

Algorithm:

Correlation,

old:

90,

Keep

responseifP3 was found at site Fz. Condition:

Monitor,

Algorithm:

Correlation,

Regressed: _yes, EOG Thresh

old: _90, CorrelationThreshold: 0.68 74

Averages.

Keep

response ifP3 wasfoundat site Fz. Condition:

Monitor,

Algorithm:

Correlation,

old:

90,

CorrelationThreshold: 0.68 75

Keep

response ifP3 wasfound at site Fz. Condition:

Monitor,

Algorithm:

Correlation,

Regressed: yes, EOG Thresh old:

90,

Correlation Threshold: 0.68 . . 75

B.27 Response Verification Trials Subject 6 GoalandNon-Goal Grand

Averages.

Keep

response ifP3was found at site Fz. Condition:

Monitor,

Algorithm:

Correlation,

old: _90, CorrelationThreshold: 0.68 76

B.28 Response Verification Trials Subject7 GoalandNon-Goal Grand Averages.

Keep

response ifP3 wasfound at site Fz. Condition:

Monitor,

Algorithm:

Correlation,

old:

90,

_{Correlation Threshold: 0.68} ₇₆

Averages.

Keep

response ifP3was found at siteFz. Condition:

Monitor,

Algorithm:

Correlation,

old:

90,

_{Correlation Threshold: 0.68} ₇₇

B.30 Response Verification Trials Subject 9 Goaland_{Non-Goal Grand}

Averages.

Keep

response if P3was found at siteFz. Condition:

Monitor,

Algorithm:

Correlation,

old:

90,

_{Correlation Threshold: 0.68}

. 77

B.31 _{Accuracies resulting from varying threshold for}_Correlation_Inde

terminate algorithm. Regressedtrue, EOG Threshold: 90 . 80

Accuracy

versus Original

Accuracy

for P3 Found and P3 Absent methods (site Fz). Condition: Moni tor, Algorithm: Correlation

Indeterminate,

_Regressed:

yes, EOG

Threshold:

90,

_{Correlation Threshold:} _0.68.

(12)

Found and P3 Absent methods (site Cz). Condition: Moni

tor,Algorithm: Correlation

Indeterminate,

Regressed: yes,EOG

Threshold:

90,

Accuracy

versusOriginal

Accuracy

for P3 Found and P3 Absent methods (site Pz). Condition: Moni

tor, Algorithm: Correlation

Indeterminate,

Regressed: _yes, EOG Threshold:

90,

Accuracy

versus Original

Accuracy

for P3 Found and P3 Absent methods (site Any). Condition: Moni

tor, Algorithm: Correlation

Indeterminate,

90,

Correlation Threshold: 0.68. 83

B.36Across All Subjects Response Verification Trials Goal and Non-Goal Grand Averages.

Keep

Monitor,

Algorithm: Correlation

Indeterminate,

Re

gressed: _yes, EOG Threshold: _90, Correlation Threshold: 0.68 84 B.37 Response Verification Trials Subject 1 GoalandNon-Goal Grand

Averages.

Keep

Monitor,

Indeterminate,

Regressed: yes,

EOG Threshold:

90,

Averages.

Keep

Monitor,

Indeterminate,

Regressed: _yes, EOG Threshold: _90, CorrelationThreshold: 0.68 85

Averages.

Keep

Monitor,

Indeterminate,

Regressed: yes,

EOG Threshold:

90,

B.40 Response Verification Trials Subject4 GoalandNon-Goal Grand Averages.

Keep

Monitor,

Indeterminate,

90,

B.41 ResponseVerificationTrials Subject 5 GoalandNon-GoalGrand Averages.

Keep

Monitor,

Indeterminate,

Regressed: yes, EOG Threshold: 90, Correlation Threshold: 0.68 . 86

Averages.

Keep

responseifP3 wasfoundat siteAny. Condition:

Monitor,

Indeterminate,

Regressed: yes,

EOG Threshold:

90,

Keep

responseifP3wasfound atsiteAny. Condition:

(13)

Averages.

Keep

P3 found Any. Condition:

Monitor,

Indeterminate,

Regressed: yes, EOGThreshold:

90,

CorrelationThreshold: 0.68

(14)

Tables

2.1 Brain

imaging

methods that canbeused for BCI [27]. 9

4.1 Changein_accuracyfrom theoriginal _accuracywhen

keeping

re

sponses with P3 signals _using the peak _picking and correlation algorithmstorecognizeP3signals. The besttheoretical_accuracy occurs if all selected goals are kept while all selected non-goals are rejected (P calculated with theWilcoxon sign ranktest). 42

4.2 Change in the bit rate from the original bit rate when

keeping

responseswith P3 signals_usingthepeak_pickingand correlation algorithmsto recognize P3s. The best theoretical _accuracy oc cursifall selected goals arekept whileall selected non-goals are

rejected. ₄₃

B.l The number of responses used in subject averages. Condition:

Monitor,

Algorithm: Peak

Pick,

90,

Peak Threshold: 61 . 58

B.2 Stimuli from original experiment broken into selected, rejected,

goal, and non-goal. Condition:

Monitor,

Algorithm: Peak

Pick,

90,

B.3 Discarded EOG: The number of stimuli notused in analysis be cause their signalswere greater than the EOG threshold. Thus

thesewere unaffected

by

response_{verification,} butwerestillused todetermine the new_accuracy after response verification. Con dition:

Monitor,

Algorithm: Peak

Pick,

Regressed: yes, EOG

Threshold: _90, PeakThreshold: 61 59

B.4 Originaland responseverification accuracies. Verification

(q)

in dicatesthe accuracyoftheresponse verificationtrialitself. Theo

reticalbest indicates best_{accuracy that}couldhave beenachieved ifall selected goalswerekeptandallselected non-goals weredis carded

during

response verification. All Pvalues werecalculated with the Wilcoxon sign rank test. Condition:

Monitor,

Algo rithm: Peak

Pick,

90,

Peak

(15)

(q)

indicates thebitsper minuteofthe

itself. Theoretical best indicates the best bit rate that could have beenachievedifallselectedgoals werekept andallselected

non-goals werediscarded

during

responseverification. Condition:

Monitor,

Algorithm: Peak

Pick,

90,

PeakThreshold: 61 60

B.6 The number of responses used in subject averages. Condition:

Monitor,

Algorithm:

Correlation,

old:

90,

B.7 Stimuli fromoriginal experiment broken into selected, rejected,

goal,andnon-goal. Condition:

Monitor,

Algorithm:

Correlation,

90,

B.8 Discarded EOG: The number of stimuli not used in analysisbe

causetheir signals were greater than the EOG threshold. Thus thesewere unaffected

by

response_{verification,} butwere stillused todetermine the new _accuracy after response verification. Con dition:

Monitor,

Algorithm:

Correlation,

90,

Correlation Threshold: 0.68 . 69

B.9 Originaland response verificationaccuracies. Verification

(q)

in dicates_{the accuracy}oftheresponseverificationtrialitself. Theo reticalbest indicates best_{accuracy that}couldhave beenachieved ifallselected goalswerekeptandallselected non-goalsweredis carded

during

responseverification. All Pvalueswerecalculated with the Wilcoxon sign rank test. Condition:

Monitor,

Algo rithm:

Correlation,

90,

Corre

lationThreshold: 0.68 ₇₀

B.10Original and response verification bits per minute. Verification

(q)

indicatesthebitsperminute oftheresponse verificationtrial itself. Theoretical best indicates the best bit rate that could have beenachieved ifallselected goalswerekeptand all selected non-goalswerediscarded

during

_{response verification.} _Condition:

Monitor,

Algorithm:

Correlation,

Regressed: _yes, _{EOG Thresh}

old: _{90, Correlation}_{Threshold: 0.68} . ₇₀

B.ll The number of responses used in subject averages. _Condition:

Monitor,

Indeterminate,

_Regressed: yes, EOG Threshold:

90,

B.12 Stimuli from original experiment broken _{into selected,} rejected

goal, and non-goal. Condition:

Monitor,

_Algorithm:

Correlation

Indeterminate,

90, Correlation

Threshold: 0.68 .

(16)

cause their signals were greater than the EOG threshold. Thus

thesewereunaffected

by

response_{verification,}butwere stillused todeterminethenew_accuracyafter response verification. Condi

tion:

Monitor,

Indeterminate,

Regressed:

yes, EOG Threshold: 90, Correlation Threshold: 0.68 80

B.14 Originaland response verification accuracies. Verification

(q)

in

dicates_{the accuracy}oftheresponse verificationtrialitself. Theo

reticalbest indicates best_{accuracy that}couldhave beenachieved ifall selectedgoalswerekept andall selected non-goals weredis

carded

during

response verification. All Pvalues were calculated

with the Wilcoxon sign rank test. Condition:

Monitor,

Algo rithm: Correlation

Indeterminate,

old:

90,

B.15 Original and response verification bits per minute. Verification

(q)

indicates the bitsper minute ofthe response verificationtrial

itself. Theoretical best indicates the best bit rate that could

have beenachieved ifallselectedgoals werekeptand allselected

non-goals werediscarded

during

response verification. Condition:

Monitor,

Indeterminate,

Regressed: yes,

(17)

(18)

A/D analog-to-digital

ABI Adaptive Brain Interface

AC alternatingcurrent

BCI brain-computer interface

CNV contingent negativevariation

EEG electroencephalogram

ECG electrocardiogram

EMG electromyogram

EOG electrooculargram

EP evoked potential

ERN error-related _negativity

ERD event-related desynchronization

ERS event-related synchronization

ERP event-related potential

fMRI functional magnetic resonance

imaging

MEG magnetoencephalography

PET positron emission _tomography RV response verification

SCP slow cortical potential

SEP _{somatosensory}evoked potential

SSAEP steady state_auditory evoked potential

SSSEP steady state_{somatosensory}evoked potential SSVEP _steady state visualevoked potential

TTD Thought Translation Device

LSP Language Support Program

VEP visualevoked potential

VR virtual _reality

(19)

Introduction

Brain-Computer Interfaces

(BCI)

are communication and control mechanisms thatdo not _rely on peripheral nervesand muscles [84]. Aswith all non-perfect information transfer mechanisms BCIs suffer from errors [71].

However,

error recoveryand correctionin BCIhas

largely

beenneglected. Thisthesisdescribes a method to improve BCI _accuracy

by

automatically recovering errors based on brain signals that occur when the computer provides feedback to a user's selection.

Amyotrophic lateral sclerosis

(ALS),

multiple sclerosis, cerebral palsy, and spinal cord

injury

are afew neuromusculardisorders thatcan_completelypara lyze individuals butnot affecttheirbrains. Ifrestorativetreatmentsareineffec tive,theafflicted_maylive_manyyears withlifesupport systems andtwenty-four hourcare, butwithout_{the ability to}communicateor control theirenvironment

through normal means. These individualsarelocked-into their bodies [84].

Depending

upon the individuals'

afflictions, there are three options avail

able for communication and control.

First,

_they _may use _remaining _voluntary musclecontrolfor communication. Someparalyzed people retain control of eye

movements, which _they can use to answer simple questions or control word processingprograms witheyetracking.

Second,

neural pathways_mayberecon nected aroundthebreakto control

healthy

muscle. Electromyographic

(EMG)

signalsfrommuscles unaffected

by

spinal cord

injury

cancontrol paralyzedmus cles. _Finally, ifmuscularmovementis_{nonexistent, the}locked-in individual_may employ a direct brain communication mechanism to communicate and control

the environment_usingacomputer [83].

The majorityof BCI research is rooted in the electrical _activity generated

by

the neurons

firing

in the

brain,

electroencephalogram

(EEG),

because this activityhas been

throughly

studiedincognitive_psychologysinceits

discovery

by

Hans Berger in 1929 [46]. In addition, EEG is fastand cost effective compared

to other brain

imaging

methods

(fMRI,

SPECT, MEG,

_etc.);

though,

_{it only} provides aggregateinformationonneuronal activity.

Over thepast two decades_{the utility} of_{BCI has been}

demonstrated

using different brain _{signals, to varying} degrees of_success, in _spelling _applications

(20)

[24,

9,

53],

cursor control

[79,

80, 84, 63],

virtual environment control

[7,

62],

games

[30,

36],

and hand orthosis and direct muscle control

by

stimulation

[55,

61].

Current BCIs are error prone and slow with error rates of 10 to 30% and transmissionrates of10-25 bits/min

[57,

83].

Comparatively,

a mediocre typist can achieve750bits/min(30words/min). ErrorrecoveryandcorrectioninBCI has

largely

beenneglectedeventhougha modest gainfromrecoverycanachieve

significantimprovements in_accuracy andbandwidth. The lackof error_recovery methods _maybe duetoBCI's

infancy

asafield andtheneglect of performance

errors incognitive_psychology

[43],

whichhasresulted ina minimalsetof error

theoriesavailablefor application inBCI.

Twotypes oferrorsoccurin _{BCI: selecting}theincorrectchoice and _missing

the correct choice

[53],

these are also referred to as false positive and false negative errors respectively. Improved signal recognition has increased BCI accuracy,

however,

high error rates persist. Some methods oferror _recovery are: a manual undo choice

[52],

_requiring the user _{to manually} validate the selectionwith response verification

(RV)

[82],

and

detecting

the Error-_Related

Negativity

(ERN) [15]

or error potential

[71]

brainsignalsforautomaticresponse

verification.

Thesetechniques havevarious disadvantages. Themanual_techniques, undo

and

RV,

slowtheoveralltransmissionspeedoftheBCIandrequire morechoices tobe ignoredwhen no errors arepresent. Theautomatic_techniques, ERNand error _potential, do not reduce the speed of the _BCI;

however,

the ERN has only been detected with physical movement (it occurs when the user realizes

a mistake was _made, such as _pressing an incorrect button). In addition, it is

unclear whattheerror potential represents and moreexperimentation isneeded toprove its robustness acrossindividualsandtasks.

For_many

BCIs,

falsepositive andfalsenegativeerrors are

diametrically

_op

posedbecause _{reducing the}numberofincorrect selectionsincreasesthenumber

ofincorrect rejections and vice versa. In most _cases, it is betterto reducethe

numberofincorrectselections attheexpense ofincorrect rejections because re

covering incorrectselections canbemore

frustrating

astheuser must correctthe error,whileincorrect rejections_onlyrequiretheusertowaitfortheappropriate

choice tobe presented again [53]. Ifautomatic error _{recovery is}employed the

number of incorrect rejections can be minimized without

frustrating

the user

because incorrectselections are recovered without user involvement.

Thisthesispresents a new error_{recovery technique that}addressesthedisad

vantagesofcurrent techniquesand the

difficulty

of_minimizingthese diametric error types. The new _{technique automatically} rejects selections when the P3 brainsignal is not presentin a response to a selection.

Conversely,

the the se lectionisacceptedif theP3signalispresentintheresponse a responseisthe feedback from a computer when the user makes aselection, e.g. in a _spelling

application, ifchoice 'b' is selected the computer displays the letter 'b'

to the user,which isthe response tothe selection.

In the second chapter the broad background required for brain-computer

(21)

presented. Different electricalsignals beneficialto BCI are overviewed andthe BCIs that utilize them are presented. Brain

imaging

in general is touched upon and EEG is discussed in depth.

Finally,

BCI errors and performance

measurement inbit/rate areexplained.

The third chapter is a review of current error _recoverystrategies. Choice organization'simpact ontheoccurrence of errorsis discussed. Manual_recovery strategies, undo and manual response _{verification,} _along with the automatic recovery strategies, using the brain signals ERN and the error potential, are presented withtheir advantagesand disadvantages in BCI.

The fourth chapter details the automatic error _recovery method _using the P3signalforresponseverificationandtheresults ofanofflineexperiment _using thismethodina_{P3 based BCI controlling}a virtual apartment. Withatheoret ical mean _{improvement in accuracy from}78% to

85%,

therewas a_{statistically} significant improvement (P <

0.008,

Wilcoxon signedrank

test)

_{in accuracy} of 3% usinga correlationalgorithmfor P3 signal detection on responses.

Thefifth chapter describesthe chess BCIplatformforexperimentation de veloped toaddress the limitations ofthe virtual _reality

(VR)

apartment offline data analysis study. The platform provides an_engaging environment that re quires _{100% accuracy} and isa novel alternative to the traditional BCIspeller applications. In addition, a collaboration with researchers at the Institute of Medical

Psychology

andBehavioral

Neurobiology

(IMPBN)

atthe

University

of Tubingen in

Tubingen,

Germany

tousethe chessBCI forone oftheir patients isreviewed.

(22)

(23)

Brain-Computer

Interfaces

(BCIs)

A brain-computer interface

(BCI)

is a communication and control mechanism

that does not _rely onperipheral nerves and muscles [84].

Fundamentally,

BCI is asystemto record functionalactivity

directly

fromthe

brain,

recognize the

activity recorded, and control a device based on _{the activity recognized,} see

Figure 2.1.

Ideally,

brain

imaging

to recordfunctional brain_activityshould be

fast,

fine

grained, and_{nonencumbering}foreffective real-time devicecontrol.

Practically,

brain

imaging

devices _today_{only partially} fulfill thesethreeconstraints.

Mag-netoencephalography

(MEG)

is fast and finegrained, but requires aroom full

of equipment andthe subjectmust remain still. Functionalmagnetic resonance

imaging

(fMRI)

is fine grained, but requires a large machine that cannot be

moved,and isslowin_recordingactivity.

Electroencephalography

(EEG)

is

fast,

inexpensive,

and the subject is allowed alimited range ofmovement,

however,

it _only records aggregate neuronal activity.

Among

_{these choices,} EEG is the de facto standard in BCI research because it is inexpensive and _{the activity}

recorded

by

EEG is backed

by

seventy-five years of research _experience, while MEGand fMRI arerelativelynew and expensive technologies.

Recognition ofbrain _activity is limited

by

the accuracy ofthe brain

imag

ing

technique used andconfounded

by

_{the cacophony}inthe brain itself.

Many

algorithmshave beenemployedtoincrease theclassification rate ofEEG BCIs. These methods include simple linearmethods, such as _{averaging EEG} signals

over multiple trials of the same stimulus

[25],

as well as correlations between individual trials and predetermined subject averages [6]. Machine

learning

al

gorithms have been used to improve BCI accuracy with techniques such as

Independent Component Analysis

(ICA)

forthe reductionofartifacts

[34]

and

Support Vector Machines

(SVMs)

for

increasing

overall_accuracy [42].

BCIs have been used to control various devices such as a hand orthosis

(24)

Electrodes

Device State

ControlDisplayState

amp Feature

Extrator

n

User-Reported Error

Device

Device Controller

Figure 2.1: General BCI frameworkproposed_byMasonandBirch [40]. Showsthe flowof information fromthebrainto thedeviceandfeedback fromthedevicetotheperson.

[55,

61]. BCI hasalsobeenusedincomputer applicationsfor_spelling

[24,

9,

_53], cursor control

[79,

80, 84, 63],

games

[30,

36],

and virtual environment control

[7,

62].

Currently

BCI islimited

by

its transmissionrate of 10-25bits/minand error rate of10-30%

[57,

83],

BCI isan

interdisciplinary

field_combiningresearchincognitive_{neuroscience,} signal _processing, and computer science. This chapter covers the broad back ground required for brain-computer interfacing. The physical and functional anatomy of the brain is described with specific focus on aspects important in EEG BCI. Brain

imaging

methods are discussed and EEG is detailed in depth

including

the benefits of different electrical signals and the BCIs that utilize them.

Finally,

_measuringBCIperformance in bit/rate isexplained.

2.1

The Physical Brain

The adult human brain contains 100 billion neurons spread through the cere

brum,

cerebellum, limbic system, andbrainstem showninFigure 2.2 [77]. The mostimportantstructuretobrain-computer interfaces

(BCIs)

is thecerebrum's outer fourmillimeters

(mm)

oftissuecalledthecerebralcortex,which contains 20 billion neurons

[77]

_{generating the} electrical _{activity that} drives the typical electroencephalogram

(EEG)

BCI.

There are _many different types of neurons in the central nervous system (brainand spinal_{cord) varying}in diameter (0.004mmto 0.1 mm), length(0.15 mm to 2 meters), and _{shape, among} other attributes. The two main neuron classes inthe cerebralcortex are pyramidal cells

(pyramid-shaped)

and stellate cells

(star-shaped)

[39,

77]. Pyramidal cells are longer than stellate cells and are oriented to form adipole layer projectingelectrical activity to the cortical surface, which is used in an EEG BCI [26]. Neurons consist of three main structures: _{the soma,}

dendrites,

and axondepicted in Figure2.3. [image:24.476.109.434.103.237.2]

(25)

Limbic

System

Figure 2.2: Brain Structures: cerebrum, cerebellum,limbicsystem, andbrainstem. Used with permission [3].

as a nucleus, mitochondria, and ribosomes. Dendrites branch from the soma

forming

a tree shape. The dendrites are the inputs into the neuron and are

covered with _many synapses, which are the input points. The axon is the

neuron's output. Unlike the branches formed

by dendrites,

the axon has a

single connectionto thesoma calledtheaxonhillock. Theaxon resembles a

long

thread with frayed ends. The ends ofthe axon form synapses with dendrites

whereinformationistransfered [39].

The dendrites and axons of different neurons _stop short of _touching each

other at the synapse.

Instead,

a _gap is formed called the synaptic cleft and

theelectricalimpulse that travels through the axon is converted toa chemical

signalthat traverses the gap and is converted backto an electrical impulse

by

the receivingdendrite [39].

Neurons are the brain's decision makers,

forming

intricate webs with each

other_givingrisetothebrainsprocessingpower. Eachneuron_mayhave 1,000to

10,000connections with neuronsin their immediatearea orfarpartsofthebrain

[4,

77]. When a neuron receivesinput an electrical potential is created, when

thispotential reachesathreshold theneuron

'fires',

_sendinganelectricalimpulse down itsaxontowardsother neurons. Aneuron canexcite otherneuronstofire orinhibitother neurons from

firing

_{using this}process.

[39,

45]

Neuroscientists have shown that groups of neurons will fire in the same location on the cerebral cortex based on function. This has given rise to the

functional map ofthe brain shown in Figure 2.4. In the functional map, the

brain is divided into four lobes:

frontal,

parietal, occipital, and temporal [39],

The frontal lobe extends from behind the eyes to the _top ofthe head and

isresponsibleforanalysis, planning,

decisions,

movement and motor skills, lan

(26)

Input from

other neurons

Electrical

impulses

V

Transmitter

release

Dendrite

Soma

-Nucleus Axon

Hillock

Dendritic

Spine

Node of

Ranvier

GlialICell

TerminalButtons

Figure 2.3: Projectionneuron_{showing information}flow. Usedwith permission [27].

to about where theskull begins_{to steeply} slope downwardsand is responsible for

body

locationandisthereceiverof_{sensory information}fromthebody. The

occipital lobe extends from the parietal lobe to the back ofthe head and sits above thecerebellum. Theoccipital lobe isresponsiblefor_processingvisualin

formationandhasadirect linkwiththeeyes.

Finally,

thetemporallobeextends

along both sides ofthe head parallel to the ears and touches all three of the

other lobes. The temporallobe is involved with speech comprehension, recog nizing objects, scenes, and

faces,

and_maintaining autobiographicalinformation

inconjunction withthe frontal lobe

[65,

37],

2.2

Measuring

Brain

Activity

There are a variety of

imaging

devices that can be used for BCI. These de

vices include electroencephalogram

(EEG),

magnetoencephalography

(MEG),

positron emission_tomography

(PET)

, andfunctionalmagneticresonance

imag

[image:26.476.160.386.97.401.2]

(27)

Frontal Lobe Parietal Lobe

vOccipital Lobe

TemporalLobe

Figure 2.4: The brain's functional lobes. Usedwith permission_[27]

Table 2.1: Brainimagingmethodsthatcanbeusedfor BCI [27].

Method Description

Electroencephalography

(EEG)

Maps general brain _{activity using scalp} electrodes.

Magnetoencephalography

(MEG)

Measures magnetic fields generated

by

electricalcurrents at celllevel. Positron Emission

Tomography

(PET)

The subject ingests radioactive tagged glucose. After the glucose enters the

bloodstreamthePET machine measures the concentrations of_glucose, which cor

respondsto thebrain's active areas. Single-Photon Emission Com

puted

Tomography

(SPECT)

Similar to

PET,

but with poorer spatial [image:27.476.80.326.119.339.2]

(28)

Method Description

Functional Magnetic Resonance Based on Magnetic Resonance

Imaging

(fMRI)

(MRI)

technology. fMRI can detect oxy

gen levels in blood to show variations in

neural_activitywithout

ingesting

radioac

tive markers.

Figure 2.5 depictsspatial resolutions for different brain

imaging

techniques.

Computer

Tomography

(CT)

scans arethe most fined_{grained, approximately}

0.3 mm to 1 mm.

Unfortunately,

CT uses X-rays toscan the

brain,

which can

damage cells. In addition, it does not provide functional information about

the brain'sactivity. EEG provides a good functional _map ofthe overall brain

activity but has poor spatial _{resolution, ranging} from 26.6 mm to 35.3 mm. Neurons'

widthsrangefrom 0.004mm (granule_{cell) to}0.1 mm (motor_neuron)

[77].

(a)CT Scan[19] (b)MRI_(33] (c)fMRI (d)SPECT[19

(e) PET_[73] _(f)PET/MRI (Multi-Modal)[81]

(g)MEG (Superimposed

MRI)[27]

(h)EEG

Figure2.5: Examplebrainscans _depictingspatial resolution. ComputerTomography(CT) isan _{X-ray imaging}technique thatdoesnotprovidefunctionalinformation. Usedwith per mission.

Figure 2.6 compares

imaging

methods'

spatial resolution to temporal res

[image:28.476.109.433.287.539.2]

(29)

requiresaroomfull ofexpensive monitoringequipmentandthe subject's head

cannot move

during

the recording. While EEG spatial resolution is poor, it

hasverygoodtemporalresolution,whichisimportant when_translating_rapidly

changingbrainactivity to

users'

desires. EEGsignalsare_widelyusedinBCIre

searchbecause _theyarefasttemporally, inexpensive andless cumbersomethan

other methods

(MEG,

fMRI, SPECT,

and PET require a room full ofequip

ment), and non-invasive (the user wearsacap ofelectrodes)

[27]

. In addition.

EEG signals can_easily be combined with other techniques, such as

fMRI,

for

BCIsystemsthatrequirefinegrainedspatial resolution. Forthesereasons most

BCIresearch,

including

thisthesis, focuses onEEG.

36 , ,.

34:

32

30

28

ct

26

10

s

4:

2 :

SPECT

fMRI

PET

EC

CT

+

11

MRI

+ 1CT 10'*

0.1 1 10 10

Temporal Resolution

(sec)

Iff 10

[image:29.476.61.347.251.566.2]

(30)

2.3

Recording

EEG

Activity

Thissectiondetails recordingEEG from the scalp, known as surface EEG. To

record EEG fourtypes of equipment are required: _electrodes, an

operational-amplifier,an analog-to-digital

(A/D)

_converter,anda computer. Theelectrodes acquire _analog electrical signals from the scalp, which are then sent to the operational-amplifierforamplification. Afteramplificationthesignalis digitized

by

the A/Dconverter andtransfered toacomputerwhere_they are interpreted in real-time _using signal _processing algorithms or stored for later processing.

In BCI

literature,

interpreting

signalsfor communication andcontrol whilethe

subject_{is using}theBCI is referredtoas _online,while_processingstored signals

is referredto as offline.

As described in Section

2.1,

whenneuronsfire_theygenerate electricalactiv

ity

thatsummatesin thescalp. Thiselectrical_activitycreatesdifferentelectrical voltages

(potentials)

on the scalp, which are detected

by

electrodes. Thevolt ageson_{the scalp} fromthe neuron

firing

isvery small, typicallyat most 50/J.V, and needs to beincreased to the_sensitivity range oftheA/D converter, which

is _usually 0.5 V to 10V before further processing.

The_scalp voltageis increased to the A/Dconverter's range with an opera tional amplifier, which is similar to an audio amplifier found in car and home theater systems. Amplifiers increase the voltage

by

a multiplicative constant called gain, as shown in Equation 2.1 where

U,

is the input _{voltage, g} is the gain, and

V0

isthe amplified output voltage.

V0

=

Vt

x_g

(2.1)

For example, a typical EEG amplifier gain is 50,000. Given an input of

10fiV the output will be 500,000 uY, whichis 0.5 V, and withinthe range of

anA/D converter.

The electrical potentials on _{the scalp} are _analog signals that need to be

convertedtodigitalrepresentationsfora computertoprocessthem.

Converting

an_analogsignaltodigital form iscalledsampling, becausethecontinuous_analog signal issampled

(recorded)

at discrete time intervals. Two important aspects ofanalogto digitalconversion are _{the sampling}rateand resolution.

The_samplingrateindicates howoftentheA/Dconverter samplesthecontin

uous signal. ThisrateisrepresentedinunitsofHertz

(Hz),

whichare1/seconds. Ifthe _samplingrateis 128

Hz,

thenthe signal issampled 128timesa second.

The fasterthe_samplingratethemore accuratethesignalwillberepresented in digital form. The Nyquist theoremstates the highest

frequency

that canbe

accuratelyreconstructedwithout erroris halfthe_samplingrate. Inotherwords, to represent an analog signal without error in digital form the signal must be

sampled at leasttwiceits

frequency

[26]. As showninSection

2.4,

brainsignals

have frequencies up to 30 Hz. To accurately represent 30 Hz beta signals in digital form theNyquist theorem statesthe signal must besampled at least at

60Hz (2 x 30Hz). Ifthebetasignalissampled atlessthen60 Hzfalse

frequency

components will appear inthe reconstructed signal. The false

frequency

error

(31)

The A/D converter's resolution also contributes to _{its accuracy in} repre

senting the signal in digital form. Resolution is given in bits a bit is the fundamentalunit of

binary

_computingand caneither bea0 or 1. A resolution

of12bitsrepresentsa signal _using4,096points (212 =

4,096),

_{a resolution}of16 bits has 65,536 points, and a resolution of 2 bits has 4points. The resolution

points are evenly spread across the A/D's voltage output range and indicate

the precision ofthe signal's recording, which is a practical consideration when

comparing signals.

For example, an A/D converter with 12 bit resolution and output range

5 V to 5 V measures the signal in 0.002 Volt increments (10 V total range,

4,096 points, 10 V/4096 =_0.002

V)

An A/D converter with 16 bit resolution and output range -5 V to 5 Vmeasures the signal in 0.0001 Volt increments

(10V/65536= _{0.0001 V). The}_choiceof resolutionis basedon the

fidelity

the BCIsignal_processing requires.

2.3.1

Electrodes

Electrodesconducttheelectrocorticalpotentialsfrom_{the scalp to} an amplifier.

Surface EEGelectrodesare 8 to 9mm indiameter and are composed of

silver-silverchloride

(Ag/AgCl),

gold, ortin [26]. Needleelectrodes aregenerally not

used for BCI as _they are uncomfortable for _{the subject, may} cause

infection,

and have poor_{recording quality} [66]. Throughout this thesis 'electrode'

refers

to surface electrodeunlessotherwisestated.

Electrodesarecup-shapedtoholdelectrolyticpasteor gel

depending

onhow

they

are affixedto thescalp. Theelectrolytic paste and gel aid_{the conductivity}

oftheelectrodes and

help

preventmotion artifact. Forgood_{quality recordings,}

theelectrodesmustbe

firmly

attachedto the scalpandtheelectrodeimpedance

should between 100 and 5,000 Q

(Ohms)

[26].

Impedance isthe resistanceto current flow. Iftheimpedance betweenthe

electrodeand _scalpis

high,

thebrain's electrical_activitywillnot beconducted

through the electrodes _properly and large differences in impedances between

electrodesfavors 60 Hznoise

[26],

seeSection 2.3.5. Electrode impedance ismea

sured

by

_sendinga weak_alternatingcurrentthroughone electrode and_recording

it fromasecond electrode. With proper electrode application impedances can

bereducedtoless than3,000 ft [64],

Only

meters_{specifically designed}_{to test scalp} electrodeimpedancesshould

beused. Impedancemeters designedto testelectrical circuits_maysend alarge

painful current _{to the patient,} and can also polarize the paste or gel _changing

their conductive properties [26].

2.3.2

Electrode

placement

The International 10-20 SystemofElectrode Placement

[32],

seeninFigure

2.7(a),

was developed in 1958

by

electroencephalographers to compare data _using a

common

terminology

and reference_system,anddefinestheplacementof

(32)

of electrodes 10% and 20% ofthe distance betweenskull landmarks [29]. The electrode position names consist of a letter followed

by

a number.

The letter indicates the structural lobe below the electrode:

Frontal, Central,

Parietal, Occipital,

and Temporal. Numbers indicate if the electrode is

left,

right, or on the midline. Odd numbers indicate

left,

even numbers indicate

right, and

'z'

for 0 indicateson the midline. Thesystem uses _every other odd

andeven number _startingwiththree andfourtoleaveroomfor additionalelec trode placements.

Forexample, from Figure

2.7(a),

01 isontheleft side ofthe head overthe

occipital

lobe,

and Cz istheposition at_{the very}_topoftheheadoverthecentral lobe onthe mediallineofthe left and right cerebralhemispheres.

Front

Left Side

-20%

10% Nasion *

20% 4

20% y y. i

i

20%

\

o

H~

/ 10%

-* _Inion

Back

(a)

(b)

Figure2.7: International10-20SystemofElectrodePlacement, _(a)Topviewshowing the21

electrode positions, _(b)Sideviewshowingelectrode positionsat10 and20%ofthedistance

betweenthenasion andinionmeasured overthetopofthehead. Usedwith permission[27].

2.3.3

Reference

and Bipolar Recordings

EEGmeasuresthepotentialdifference

(voltage)

betweentwo electrodes,i.e. the

electrical signal from one electrode is subtracted from the other. In a bipolar

recording, the two electrodes both measure cortical activity. In a reference

recording, oneelectrode measures corticalactivitywhile theother is placed on

[image:32.476.114.442.261.468.2]

(33)

reference all electrodesmeasuringcortical_activityarelinkedtoasingle reference

electrode [64].

2.3.4

Grounding

Equipment and subjects must be _properly grounded

during

EEG recordings

for safety andto reduce noise [64]. A

bony

protuberance, such as themastoid

behind the ear, should be connected with an electrode to the EEG amplifiers

electrode groundjack.

2.3.5

Artifacts

Artifacts are non-cerebral activity inthe EEG _{that may} masquerade asbrain

activity and otherwise obscure realbrain activity. Artifacts appear becauseof

the large amplification required to measure electrical brain activity, and _may

betechnological or physiologicalinnature. The

following

isalist ofthemajor

artifactsthat mayoccur when_performingBCIexperiments [66].

Technical artifacts

Electrode

Improperly

attached electrodes and

faulty

wires can cause

artifacts. An impedance checker shouldbe used _{to verify that}electrodes

are_properly attached. Electrode impedancesshouldbe lessthan5,000fl

[64].

Faulty

and _partially broken wires are difficult to diagnose as _they _may

intermittently

causeartifacts.

Faulty

wires_maybe identified

by looking

at

the activity in nearby electrodes;because EEGmeasuresthepotentialfield

across the

head,

electrodes near each other should show similar _activity,

dissimilar _{activity may}be a result oftechnical artifacts [64],

60 Hz interference

Nearby

electrical equipment and power lines can

induce rhythmic 60 Hz cycles (50 Hz in

Europe)

from _alternating cur

rent

(AC)

power. _{Proper grounding} of the subject and equipment can

reduce this interference. In addition, amplifiers

typically

have 60/50 Hz notch filtersthat removethisnoise before amplification [64].

Ground

loop

- 60

Hz interference thatoccurswhen a ground electrode is shorted to an active electrode [64].

Physiological artifacts

Oculography - Eye

movement and

blinking

cangenerate spikes of200 to

400 fiV, whichtravel posteriorly. Eye electrical _{activity recordings,}

elec-trooculargram

(EOG),

areusedtodifferentiateeye _activityfromcerebral

activity. EOG is

typically

recorded from electrodes above and below the

(34)

experimenters will_generallyrejecttrialswithamplitudes greaterthan 50-90 fiV when _{classifying activity}to account for electrooculargram

(EOG)

artifact.

Alternatively,

regression algorithms _{may be}used to remove eye

artifacts fromtrials that wouldotherwise be rejected [72].

Muscle

-Musclecontractions,suchas_talking,jawclenching, andchewing,

generate electrical _activity, which _may be seen in EEG [26]. Electrical

muscle _activity,elect romyogram

(EMG),

is dominant inthe 50to 150 Hz

frequency

range and peaks at 5 mV

[38,

18]. A lowpass filter is used to

reduce muscle artifacts if thecerebral_activityof_{interest is low}enough.

Cardiac _{Electrical activity from} the

heart,

electrocardiogram

(ECG),

can appearin EEG. Pulse waves caused

by

electrodes over blood vessels

andballistocardiographicartifact caused

by

a subject's_rockingmovement when theheart beatsmayappear in theEEG [64].

Perspiration Salt in perspiration creates a "salt bridge''

between elec trodes that _{decreases inter-electrode impedances causing}shorts. Perspi

ration _mayalso separatetheelectrode from_{the scalp} overtime

[66,

26].

Motion Head and

body

movement _may move the leads and _contacts,

which causes wide wave artifacts. This can be corrected

by

_asking the

subject not to move

[26,

66].

2.4

Electroencephalogram

(EEG)

Electroencephalogram

(EEG)

is the recording of electrical _activity from the

brain,

discovered in humans

by

Hans Berger

(1873-1941)

in 1929 [46]. The

electrical_activity generated from neurons

firing,

Section

2.1,

summateson the human scalp.

Using

EEG _{recording equipment,} Section _2.3, this electrical ac

tivity

canberecognized andutilized inbrain-computer interfaces (BCIs). This

sectionreviews the signalsgenerated

by

the

brain,

where _theyare recorded on

the scalp, theirfunctionalsignificance, and theiruseinBCI.

2.4.1

Rhythmic Brain

Activity

Rhythmic brain waves were the first brain signals discovered

by

Hans Berger

[46]

. Rhythmic brainwavesarebrainsignalsthatoccurcontinuouslyand repeat

inamplitude,

frequency,

andwaveform [26]. Theserhythmicsignals are divided

into

frequency

ranges named after Greek letters: delta

5,

theta

8,

alpha a, and

beta j3, see Figure 2.8. _{The mapping} of Greek letters and

frequency

ranges

followstheir chronological

discovery,

not alogical

increasing

in frequency.

Alpha

The alpha rhythm ranges between 8 and 13 Hz in normal adults and is dis

(35)

wave-1 4 8 13 30

Frequency

(Hz)

I

I I

I

I I I

I

I I I I

I

I I I I I I I I I I I I I I I I

I

alpha beta EEG Rhythm

Figure 2.8: _{Delta, Theta, Alpha,} and Beta rhythmic EEG _frequencyranges. Usedwith permission [27].

formsaremonomorphic (regular_{in shape)} with _sharppointsatthe_top or bot

tom, or sinusoidal. Alpha's amplitudeis variable, but averages50 /jM [26]. Alpha activityis

typically

constant in its

frequency

in individual subjects,

but varies acrosssubjects, and maydecrease

by

1 Hz or more with drowsiness. It increasesforashortdurationaftereye closure,andisblocked

by

eyeopening, i.e. its amplitude _{significantly} decreases. Its

frequency

should be the same in both hemispheres at _any given _time, while the amplitude _may differ between

the twohemispheres [26].

Thealpha rhythm's purposeisunknown; though,theposterior

distribution,

eye_openingand_{closing influence}on_{the wave,}andothercharacteristicsindicate it is "integratedwiththevisualsystemfunctionand_possiblyrepresents_activity

which appears inabsenceofspecific inputto that system [26]."

The rhythm is named 'alpha' for _purely historical reasons. When Hans Burger made hismeasurements he namedthe first rhythmhe identified 'alpha'

afterthe first letter inthe Greekalphabet: a [46]. Beta

Thebetarhythm's

frequency

band is between 13and30 Hzand appearsinthree

maintypes

frontal,

_widespread, and posterior which_varyin distribution and reactivity. These three beta patterns disappear in sleep, but frontal and widespread _activity remain longer than alphaactivity

during

drowsinesswhen beta becomes more dominant. Beta rhythm amplitude is

typically

lower than

alphaamplitude, seldom_{exceeding 30}uN[26]. _{Beta activity}greaterthan30 Hz isoften referredtoas gammaactivity.

The most commontypeofbetarhythm is distributed

frontally

and extends into the central regions. This beta rhythm _may be blocked

by

movement, in

tention to move, andtactilestimulation [26].

Widespread beta rhythm appears over the _majorityofthe head and is not blocked

by

any stimulus [26].

Posteriorbetarhythmisalsoknownasfastalpha variantbecause it islocated

inthesame area asthealpharhythm,isblockedthesame as alpha_rhythms,and

intermixes,

alternates, and replaces alpha rhythms. In _addition, its

frequency

is usually twice alpha's

frequency

(16-20

Hz)

[26].

(36)

Mu

The mu rhythm's

frequency

band is between 7 and 11 Hz and appears in the

central and centroparietal regions as arch-shaped trains

lasting

afew seconds.

Mu rhythms are more apparent

during

alpha

blocking

because mu and alpha

rhythms_{overlap in}

frequency

range [26].

Similar to beta rhythms, the mu rhythm is blocked

by

both voluntary and

involuntary

movement, intentionto move, andtactile stimulation [26],

Themurhythm's purposeisunclear, but _"mayberelated_{to somatosensory}

processes associatedwithmovement [26]." The

blocking

effect caused

by

move

mentandtactilestimulation_mayindicatethemu rhythmrepresents "the

idling

ofa_sensory system not_processingspecificinput fromthe thalamicnuclei [26]."

Delta and Theta

The delta rhythm's

frequency

band is below 4

Hz;

theta's is between 4 and

8 Hz. Both rhythms _primarily occur in

deep

sleep

[26]

making them oflittle

use in brain-computer

interfaces;

_though, theta rhythms do appear in small

unorganizedamounts innormal adults

[26,

37],

BCIs based on Rhythmic

Activity

Wolpawet al.

[84]

developedaBCI_usingself-regulation ofmuandbetarhythm

amplitude. After 5-10 half hour_trainingsessionssubjects can learntoincrease

anddecreasetheamplitudeofthemu orbetarhythmtocontrolamousecursor,

and with sufficient

training

can achieve information transfer rates up to 25 bits/min.

Pineda et al.

[62]

developed a BCI based on the difference of mu power

at C3 and C4 _using Cz as ground. Users were trained for 10 hours over five

weeks togenerate muactivity under both C3 and C4 to move left ina virtual

environment, andtogeneratelessmuinonehemispherethan theothertomove

the virtual character right. Transfer rates were not reported, but all subjects

wereable todemonstrate bothtypes ofcontrol.

Brainball isanovelgamedeveloped

by

HjelmandBrowall

[30]

where aplayer

attempts to roll a ball into an opponent's goal

by

_achieving a higher state of

relaxation measuredas the ratio of alpha to betawaves. Alphaand beta

self-regulationsare rarelyusedin BCIs because_theyrequire_shiftingbetweenstates

of relaxation and alertness. These waves have _{traditionally} been restricted to

neurofeedback/biofeedback applications [1].

Doherty

et al.

[22,

21,

23]

studied the

brain-body

interface Cyberlink

[2],

whichuses acombinationofelectromyogram

(EMG), EOG,

and alpha andbeta EEG for _controlling various interfaces.

They

reported subjects withtraumatic

brain injuries achieved _accuracy rates between 44 and 100%. Though it is

uncleartowhatdegree Cyberlink's controlis

directly

fromthe brain.

Guger et al.

[28]

reported a large scale imagined motor movement BCI

experimentwith ninety-nine

healthy

subjects. After 20-30minutes of

training,

(37)

their right hand (which extended an on screen bar right) or both feet (which

extended an onscreenbar left).

2.4.2 Event Related

Potential

(ERP)

Event related potentials

(ERP)

are changes in the brain's electrical _activity in response to stimuli. Unlikerhythmic brain activity, ERPs are short signals

withdefinitive beginnings and endings. These signals are alsotime-lockedtoa stimulus,whichcanbeauditory,visual,orsomatosensory(touch). The

following

isnot an exhaustivelist of

ERPs,

butalist ofERPs thathaveutility in BCI.

P300

TheP300 isalarge brainsignalthatisevoked

by

novel andtaskrelevant stimuli. These formsaredesignated P3a

(novelty)

andP3b (taskrelevance). The P300's

name reflects that it is a positivesignal that peaks 300 ms after the onset of stimulus. Most time related evoked potentials are named in thismanner [45]. The P300 isalso referredto as theP3.

N1

Figure 2.9: P300waveform.

The P3aappears

frontally

whenthesubjectisaware ofa novel orinfrequent

stimulus. For _example, if the subject is presented with _many low

frequency

tones intermixed with infrequent high

frequency

tones (e.g. lows tones occur five times as often as high _tones), the high tones will evoke the P300 [45],

P3aexperiments are often called oddballexperiments_referringto theinfrequent

stimuli

being

rare and different from frequent stimuli.

The P3b appears _parietally when the subject is aware of a task relevant

stimulus. For _example, ifthe subject _{is repeatedly} presented letters of the alphabet oneletter at atimeand is given a_spelling_task, the P300 willoccur

whenthe current letterthesubject wantsto'type'

[image:37.476.105.297.313.478.2]

(38)

Bennington

[8]

demonstratedthe P300'samplitudeissmaller whenthe sub

ject ispassivelyaware ofthestimulusvs_{actively attending to the}stimulus(such

as _countingwhen the stimulus_{appears) in}oddball _auditory and visual experi

ments. The P300's amplitude is alsosmaller and _may disappear if the subject

is drowsy. The

latency

ofthe P300 increases with oldersubjects [45].

Thesourcesiteforthe P300in thebrain isunknown;

however,

it isthought

tobe generatedin the middle ofthe temporallobe

(medial-temporal);

_though,

thisisunproven. Becausethe P300 isevoked

by

_manydifferent _{tasks there may}

not beone source generator [45].

Steady

State Visual Evoked Potential

(SSVEP)

Steady

state visual evoked potentials

(SSVEP)

arerhythmic waves that occur

occipitallywhenthe subject is

fixating

on a

flashing

stimulus. Thewaves have

thesame

frequency

astherepetition rate ofthe

flashing

stimulus. SSVEPsoccur

whenthe stimulus flashes 10 timesper second and _{up to}62 timesper second.

Flashing

beyond 62 times persecond exceeds the average critical

frequency

of photic

driving

(CFPD)

[45].

Contingent Negative Variation

(CNV)

The contingent negative variation

(CNV)

signal occurs when a subject is re

quired to respond to an imperative stimulus that occurs

following

a _warning

stimulus, i.e. itoccurs for expectation [45],

Typically,

the warningandimperativestimulus are separated

by

_{1-2 seconds,}

and the contingent negative variation

(CNV)

begins 400 ms after _{the warning}

stimulus, peaking up to -50/iV at 800ms [45].

Thesubject must expect andintend to respond to the imperative stimulus

forthe CNVtooccur [45].

BCIs based on ERPs

The P300's robustness across multiple individuals and the lack of _trainingre

quired to utilize it makes it a desirable signal for BCIs. Donchin et al.

[24]

used theP300 ina_spellingBCI. Subjectswerepresented a six-by-six matrix of

<