UNDERSTANDING DRIVER BEHAVIOUR RELATIONSHIP TO PRECURSOR EMOTION BY USING EEG SIGNALS

(1)

UNDERSTANDING DRIVER BEHAVIOUR

RELATIONSHIP TO PRECURSOR EMOTION BY

USING EEG SIGNALS

BY

NORZALIZA MD NOR

A thesis submitted in fulfilment of the requirements for

the degree of Master of Computer Science

Kulliyyah of Information and Communication

Technology

International Islamic University

Malaysia

(2)

ABSTRACT

Driver behaviour is indeed reckoned to be one of the highest factors affecting fatal accidents. However, majority of the cases can be avoided if the driver can remain focus and make a correct decision in controlling the vehicle while driving. Decision-making ability of the driver is impeded due to driver behaviour which may involve precursor emotion of the driver that could lead to fatal accident. Thus, understanding and analyzing the driver behaviour and the resulting emotion can help prevent accident and reducing accident fatality rate. In this thesis, the correlation between precursor emotions to pre-post accidents using driving simulator is studied in details. This correlation between driver's behaviour and their respective emotion can be analysed based on the 2-D Affective Space Model (ASM) using four basic emotions (happy, calm, fear and sad) as stimuli. In this case, the Electroencephalogram (EEG) device is used to extract brain waves signal while the driver is driving the simulator. The EEG signals are captured through the scalp of the driver and features are extracted using Mel Frequency Cepstral Coefficient (MFCC) and Kernel Density Estimation (KDE). Neural network classifier of Multilayer Perceptron (MLP) and fuzzy neural network classifier of Adaptive Network-based Fuzzy Inference System (ANFIS) are used to classify discrete class emotions and the valence and arousal axes for the ASM. In the discrete class, result shows the possibility using the research method to identify the basic emotion is successful. Analysis of the precursor emotion for pre-post accidents using the driving simulator shows an interesting finding that complements the discrete classification. In addition, the analysis also indicates how precursor emotion can affect driver behaviour in pre-post accidents. Consequently, the understanding of pre-cursor emotion and its relationship towards driver behaviour could help the driver to control his/her emotions while driving which can prevent to fatal accident.

(3)

J)}l .: .. dy-

~

-)1

~\....~1 j--~....!1

..1.>-i

y.

~WI

..::-1..,{)1

JJL..

~}--

0\

~

J\)JI

i:i~ ,;~L,AJI

l)

jJUI

.%

bl ..: ..

dyl-1 oh

~

:.r< ·..'

.;_lb

t'J .a...,}l

..::-\j~

Jl.J

Jj jJUI ..1..:.>- JI)JI )lil

J>-

oJ.li!\) .o)-:-..JI

.J

,...~

-,.JL..a.!l

_yi)

,#UI ~}-- ~ D)jjJ

.a...,r

..;..,~~..:>-

Jl )_,;.;

-)1 4)\....A;'JII jJUI a...,}l

..::,~~.J-1

t"J

J ...

_,~ ..::,~i_,:l-1 ,y

J_lo·ll

,p

..lYl...: .:Jj)

,p

~jll

._;l,iy.JI

....Abiy.JI 0;.:

~1)1

'..!')WI

..._~~

-

~~

,y

ih

.J-

:::...1)

.L~.>.

_yi

er~) .o\.)\$.1

j4.>--

il~~ .:Jj)J ,..;_,~\.;LI (jJ.}! jJUI ..::-\jriJ ,;~1

'(2-D )

.j1WI Jls!;i C)_,i

J>-

~~ o~l j_,.f'~~

,#UI

~_,L

0;.: '..!')WI

~y. J)-1) ,j_,J.~ '>)_u~ ,;~L...JI .,/'J

,-...,J

~I ~\....,~I ._;l,iy.JI <.J"L,AJ .:Jj))

[~

'Ji

J~ft-<JI

t_L..UI

~

j4.>--

i~

<.f'\,.W) .jJUJ

u\#

.._A1iy.JI oh

.1\A:JI

{J ,;\.)\}.\

j4.>--

i~ jJUI Jy<.., \....1..:.>- .:Jj)J ~J\,;.~1 t_L..UI ..::-L>.._, ...

y ..

v

i~~~

,..:;; ""4W1

C~l ~~

vi)l oJ)

,y

.j\...UI

~I

..::-~\.;.~

j_,~l ~ i~~~

t

w-

.'-il!..<ll

J.J...i.:J

ypS"

vi.,io) ,)~_r.11

vL,.ii

JIH

~(.;

ri.b

J>-

~\.AJI '(..$."').:.11 ~I j_,~l ~) ,..::-~1 ~..lo.;:,. ~\

.J

o}~~~ J)l$.. -s..l?J ,_,til.(.;]\ vYJ ,jbiy.JI ..::-\!i ~

l)

\,s.:!).:.il

J'Jf.J.::...-,'Jfl

..:_,.,j\

;;,A.,.k

i\~1 a...J~I 0\ ~l.:JI ..::-.I~\ •I ai·, :ill

J., .

;I~WI l\}.1 ;>").i

. ~F ~ ~ I..:...- ~ .. J ~ """' I..:...-...;

~

\.. :ib-)

o~l

._;l,iy.JI

_yi

.Ri

L.S'

,~L; ~_,.b \~\....~\ ._;1,~....!1

-s..l>..:l

Jl

'..1\.,0~~)

,ti..a..i.J.I

..::-~1

J7<:J

4'

~l;:j

\o\.)\$.1

j4.>--

i\~~ ..;_,~\.;L\

;;b.

r

l)

JJUI

~

_,L

J>-

;\!J

o~l

4..J.WI j(

Jl

D)

..iS' ...,

l,.b.:ll ) ...

J

..W .:Jj) jJUI ..::-1_;~ i.p)l.y) ,;~1 ~iy.JI ~ J~ .:Jj)

J>-

~~) ,..;..,)\.;L\ ~ \..

..;.., ) i_,:l-I

C!

Jl

DJ)

,p)!.) , o ~

l,AJ

I ..1..:.>-

"-'.b iy

J>-

o)o..J I l) jJU I ..lY L...; j

i

:.rV. ..

. a...,)\

(4)

APPROVAL PAGE

I certify that I have supervised and read this study and that in my opinion, it confirms to acceptable standards of scholarly presentation and is fully adequate, in scope and quality, as a thesis for the degree of Master of Computer Science.

t\)'t..).Y-Abdul Wahab t\)'t..).Y-Abdul Rahman Supervisor

Hariyati Shahrima Abdul Majid Co-Supervisor

I certify that I have read this study and that in my opinion it confirms to acceptable standards of scholarly presentation and is fully adequate, in scope and quality, as a dissertation for the degree of Master of Computer Science.

lmad Fakhri Internal Examiner

OngYew Soon External Examiner

This thesis was submitted to the Department of Computer Science and is accepted as a fulfilment of the requirement for the degree of Master ofCorr:!_uznce.

z~lk~fi;·~~-~~i

... _ ... .

Head, Department of Computer Science

This thesis was submitted to the Kulliyyah of Information Technology and is accepted as a fulfilment of the require

and Communication gree of Master of Computer Science.

(5)

DECLARATION

I hereby declare that this thesis is the result of my own investigations, except where otherwise stated. I also declare that it has not been previously or concurrently submitted as a whole for any other degrees at IIUM or other institutions.

Norzaliza Md Nor

Signature ...

~.:

... . Date ... ..

~~~

j::J.O\'J.

(6)

INTERNATIONAL ISLAMIC UNIVERSITY MALAYSIA

DECLARATION OF COPYRIGHT AND AFFIRMATION

OF FAIR USE OF UNPUBLISHED RESEARCH

UNDERSTANDING DRIVER BEHAVIOUR RELATIONSHIP TO PRECURSOR EMOTION BY USING EEG SIGNALS

I hereby affirm that the International Islamic University Malaysia (HUM) hold all rights in the copyright of this Work and henceforth any reproduction or use in any fonn or by means whatsoever is prohibited without the written consent of HUM. No part of this unpublished research may be reproduced, stored in a retrieval system, or transmitted, in any fonn or by means, electronic, mechanical, photocopying, recording or otherwise without prior written permission of the copyright holder.

Affirmed by Norzaliza Md Nor.

...

~·

... .

..~!~.!:::~~?:

...

Signature Date

(7)

ACKNOWLEDGEMEN1'S

In the name of Allah, the Most Gracious and the Most Merciful, Peace be upon the Holy Prophet Muhammad (S.A.W) who has given light to mankind and His family. My praise is to Allah (S.A.W) for providing His blessing; guidance and inner strength which has helped me get through this challenging journey until successful completion of this thesis. I would like to express my gratitude to all those who have supported and assisted me along the way in completing this thesis. This thesis bears on imprint of many peoples. Alhamdulillah, I manage to complete the thesis as it has been fully supported by Prof. Dr. Abdul Wahab Abdul Rahman as main supervisor, Assoc. Prof. Dr. Hariyati Shahrima Abdul Majid as co-supervisor, and finally, Asst. Prof. Dr. Amelia Ritahani, as post-viva supervisor. Thank you for their guidance, patient, motivation, passion, massive knowledge and continuous advices throughout this journey. Next, I would like to extend my appreciation to all undergraduate students in International Islamic University of Malaysia who have participated in the experiment for their valuable time and kind assistance. I wish to express my gratefulness to my colleagues in the Pervasive Computing Brain Development Group (P-COMBAD) who rendered their help during the period of my research work. Most important of all, my deepest appreciation goes to my best friend, Mohd Khairul Anuar Atdenan who has always been assisting me throughout my studies. Last but not least, I wish to avail myself of this opportunity, express a sense of gratitude and love to my beloved auntie, Rahmah Sarib, parents and friends for their manual support, strength, help and for everything.

(8)

LIST OF TABLES

Table No. Page No.

1.1 Electroencephalogram (EEG) electrode's definition 12 2.1 Driving Simulator and Driver Behaviour 27 2.2 Research work on emotions recognition by using EEG 31 2.3 Research work on Valence and Arousal 33

3.1 Training parameters for MLP 46

3.2 Expected output based on Discrete Class (DC) Analysis 49 3.3 Method of emotion's verification (EV), emotion's 51

identification (EI), driver's verification (DV) and driver's identification

3.4 Expected output for valence arousal approach (V AA) 55 3.5 Method of memory test (MT), homogeneous (HM), 56

heterogeneous (HT), and generalization (G)

4.1 Experimental design for driver behaviour understanding 66 4.2 Emotion data slicing for Homogenous, Training data 67

(TRAIN or TRAIN A) and Testing data (TEST or TEST B) by using 5 fold

4.3 Emotion data slicing for Heterogeneous, Training data 68 (TRAIN or TRAIN A) and Testing data (TEST or TEST B) by using 5 fold

5.1 Confusion matrix, average accuracy of emotion 77 identification (VA) for memory test, driver I (happy)

5.2 Confusion matrix, average accuracy of emotion 77 identification (VA) for 5 fold test, driver 1 (happy)

5.3 Confusion matrix, average accuracy of emotion (VA) for 78 5 fold test, driver 5 (calm)

5.4 Confusion matrix, average accuracy of emotion (VA) for 78 5 fold test, driver 9 (fear)

(12)

5.5 Emotion distribution for all drivers based on memory test 80

5.6 Label and description for dynamic emotion 85

5.7 Pre-post accidents driver 1 and driver 2, the horizontal 86 black line represents the accidents occurrence, blue line represents valence (V) and redline represents arousal (A) 5.8 Pre-post accidents driver 3 and driver 4, the horizontal 87

black line represents the accidents occurrence, blue line represents valence (V) and redline represents arousal (A) 5.9 Pre-post accidents driver 5 and driver 6, the horizontal 88

black line represents the accidents occurrence, blue line represents valence (V) and redline represents arousal (A)

5.12 DASS Severity Ratings for each severity label 93

5.13 DASS results for Stress, Anxiety and Depression 94

6.1 Comparison between Objective and Conclusion 98

6.2 Comparison between hypothesis and conclusion 99

(13)

LIST OF FIGURES

Figure No. Page No.

1.1 Elementary architecture of the driver cognitive system 4 1.2 Brain model for emotion recognition 9

1.3 The affective space model with the different position of 10 basic emotions with emotion primitives axis x for valance, and y for arousal.

1.4 Electrode placement 13

3.1 Proposed research methodology block diagram for 37 driver behaviour analysis

3.2 Comparison between original data and decimated data 40 3.3 Conventional MFCC extraction algorithm 41 3.4 10 features extracted emotion data using MFCC with I 0 42

different colours representing different features

3.5 Distribution ofkemel density estimation 44 3.6 MLP network with two hidden layer 45 3. 7 Smooth signal by using 0.2 'rloess'. The blue line 48

represents valence and the red line represents arousal

3.8 Accuracy of driver's verification by using discrete class 52 3.9 Accuracy of driver's identification by using multi- 53

discrete class

3.10 Accuracy of emotion verification by using discrete class 53 3 .II Accuracy of emotion identification by using discrete 54

class

3.12 Method of precursor emotion analysis 57

3.13 Method of pre-post accidents analysis 57

4.1 Experimental research design 61

(14)

4.2 Training data block diagram for generating network classifier

65

4.3 Testing data block diagram for Memory Test, 70

Homogeneous, Heterogeneous, and Generalization

4.4 Testing data block diagram for precursor emotion 71

4.5 Testing data block diagram for pre-post accidents 72

5.1 Accuracy of emotion based on MFCC 75

5.2 Accuracy of emotion based on KDE 75

5.3 Accuracy of emotion based on 5 fold test using 76

homogenous test

5.4 Heterogeneous analysis based on 5 fold test using MLP 81

5.5 Generalization analysis (blind test using MLP) 82

5.6 Comparison of average accuracy level for memory test, 91

homogenous, heterogeneous and generalization by using ANFIS

(15)

CHAPTER!

INTRODUCTION

Accidents take place for many reasons. Yet, most accidents are caused by human error which possibility oftranspire depends on the driver's behavior and uncontrolled emotions. In particular, this thesis aims to understand the driver behavior relationship to precursor emotion by using Electroencephalogram (EEG) signals. To achieve the aim, we proposed to analyze the precursor emotions and pre-post accidents based on affective space model (ASM) which allows the emotion to be represented in valance (V) and arousal (A).

This chapter is organised as follows: Section 1.1 will cover the introduction of the thesis including the driver behaviour and driving simulator. Following this, background of the study is discussed in section 1.2 and problem statement in

section 1.3. Then, scope of the study is described in section 1.4 whereas research objectives in section 1.5. Next, seven research questions are provided in section 1.6, followed by 7 research hypotheses in section 1.7. In section 1.8, the significance of the research is explained in two subsections which consist of contribution to human life and contribution to computer science field. Research methodology is exposed in

section 1.9 and organisation of the thesis in section 1.10. Finally, summary of the chapter is covered in section 1.11.

(16)

(17)

(18)

(19)

(20)

(21)

(22)

(23)

(24)

UNDERSTANDING DRIVER BEHAVIOUR RELATIONSHIP TO PRECURSOR EMOTION BY USING EEG SIGNALS