INTERNETWORKING INDONESIA JOURNAL 53

(1)

brain-computer communication, but their classification accuracy is limited by unforeseeable signal variations due to artifacts or recognizer-subject feedback. In this paper, we propose an emotion classification method based on backpropagation neural networks to classify the EEG-SSVEP signals which are induced by the four short movie stimuli. The average of 75% classification accuracy is achieved for sad, angry, happy, and calm emotions. The evidences that higher accuracy of the EEG emotion recognition system can be achieved with higher precision of the psychological emotion design is provided.

Index Terms— Brain computer interface, Classification accuracy, Emotion recognition, Backpropagation neural networks, EEG-SSVEP

I. INTRODUCTION

rain Computer Interface (BCI) a type of communication system that translates brain activity into commands, enabling the user to control special computer applications or other devices simply by means of his or her thoughts. Signals from the brain are acquired by electrodes on the scalp and processed to extract specific features that reflect the user’s intentions. These features then are translated into commands that operate the application or device. There must be developed and maintained a good correlation between the user’s intent and the signal features employed by the BCI; and the BCI must select and extract features that the user can control and must translate those features into device commands correctly and efficiently. For this, brain activity must be monitored. Today there exist various techniques by which to accomplish this [1-10]. Among these methods, almost all BCIs reported to date have been based on EEG. The amplitudes of EEG signals can vary between 10 and 100 V [10].

There are two main approaches to detection of user’s commands from EEG. In the first approach the subject concentrates on a few mental tasks. Concentrations on each

Manuscript received October 22, 2016.

Arjon Turnip, Artha Ivonita Simbolon and M. Faizal Amri are with Technical Implementation Unit for Instrumentation Development, Indonesian Institute of Sciences, Indonesia Bandung Gedung 30, Jl. Sangkuriang, Bandung, Indonesia (emails: [email protected] and [email protected])

mental task produces a different EEG pattern. The BCI (or precisely the classifier) can then be trained to classify those patterns. Several BCIs (e.g. [1-9]) are based on this kind of pattern recognition approach. In the second approach the user has to learn to self-regulate his or her EEG responses, for example to change the rhythm amplitude [11]. Electrical activity is continuous in every living human’s brain. Much of the time, brain waves are irregular and no general pattern can be observed [15]. Helpfully, there are various properties in EEG that can be used as a bases for BCI such as rhythmic brain activity (i.e., delta, theta, alpha, and beta) [10], event-related potentials (ERPs), event-event-related desynchronization (ERD) and event-related synchronization (ERS) [1][12][13]. However, in the present study, we focused on the use of the ERP properties. ERPs refer to the potential changes in EEG that occur in response to a particular event or a stimulus.

The most commonly utilized ERP is steady state visually evoked potentials (SSVEP). SSVEP are signals that are natural responses to visual stimulation at specific frequencies. When the retina is excited by a visual stimulus ranging from 3.5 Hz to 75 Hz, [14] the brain generates electrical activity at the same (or multiples of) frequency of the visual stimulus. For this paradigm, the subject is told to concentrate to a flashed stimulus with a certain frequency. The presence, magnitude, topography, frequency, and time of the response signal are often used as metrics of cognitive function in decision making processes.

Emotion is an overall performance of human’s consciousness, which can significantly affect human’s action towards peripheral environment and thus it always plays an important role in our daily lives especially in human-human interaction. It is relatively easier for people to recognize others’ emotional states. However, how to detect and model users’ emotional states with advanced artificial intelligence techniques is a challenging topic within the human-machine interaction community. Automatic emotion recognition is one of the most challenging tasks. To detect emotion from nonstationary EEG signals, a sophisticated learning algorithm that can represent high-level abstraction is required. This study proposes the utilization of a backpropagation neural networks (BPNN) to discover unknown feature correlation between input signals that is crucial for the emotion recognition. The BPNN is implemented with a stacked auto encoder using

Backpropagation Neural Networks Training for

EEG-SSVEP Classification of Emotion

Recognition

Arjon Turnip, Member, IEEE, Artha Ivonita Simbolon, M. Faizal Amri, Poltak Sihombing, Roby

Harlen Setiadi and Edi Mulyana

(2)

hierarchical feature learning approach.

The structure of the paper is as follows. In Section 2, the subject population, the experiments that were conducted, and the methods used for data preprocessing are described. Classification using the BPNN model is explained in Section 3. Results and discussions are presented in Section 4. Conclusions are drawn in Section 5.

II. METHOD

A. Data Acquisition

In order to evoke emotions of subjects, we chose several 4-short movie clips (nature, funny, bully, and sad movies) as stimuli of four emotions (calm, happy, angry, and sad). All the movies were in English and easy to understand. In the experiment, nine subjects (all males, age 25 ± 1 years, none of whom had any known neurological deficits) have participated. The experiments were performed in the day time and subjects were asked to have adequate sleep in the day before experiment. The subjects are asked to concentrate on the four short movie. A 5-channel electrode cap according to the extended international 10-20 System at F7, T5, O1, Fz, and O2 were used to record EEG data and digitized at a 500 Hz sampling rate. The four movie clips were played (30s, 198s, 42s, 90, respectively) with a 5s rest between two clips. Each subject records four sessions to indicate four different emotions.

B. Signal Processing

Because of the measured EEG signals contain a lot of artifacts, such as eye blinks, movements, and muscle noise, the signals are necessary to perform a preprocessing step to remove the artifacts. Many algorithms have been widely used to solve this problem [15-18]. Therefore, we adapted high pass filter (HPF), band pass filter (BPF), and removal baseline. To remove the linear trends and DC components, the HPF and removal baseline were used, respectively. Furthermore, EEG signals were filtered by band pass filter between 3Hz and 30Hz where have the more significant signals than the other bands. The problem is that, there is no current standard platform on which to analyze the specific frequency data from experiments using EEG recordings. The data can be analyzed more effectively if there is a way for analyzing the data from user specified channels. Therefore the collected EEG data in the time domain need to be visualized in the frequency domain using FFT method. The feature of amplitude and frequency of each channel are used as an input to the classifier. The algorithm also allows researchers to specify channel and time interval information in order to obtain a more specific analysis of the specific stimulus.

III. BACKPROPAGATIONNEURALNETWORKS

CLASSIFIER

Artificial neural networks have been proposed in the fields of information and neural sciences following research into the

mechanisms and structures of the brain. This has led to the development of new computational models for solving complex problems such as pattern recognition, rapid information processing, learning and adaptation, classification, identification and modeling, speech, vision and control systems [3][4][19][20]. The network architecture includes statistical and dynamical, single and multilayer as well as feedback (recurrent) networks has been presented. One of the most difficult problems, and still of great interest to scientists, is learning. Special attention has been paid to the most efficient learning algorithm for multilayer networks, namely backpropagation.

The backpropagation algorithm allows exponential acquisition of input-output mapping knowledge within multilayer networks. If a pattern is submitted and its classification is determined to be erroneous, the current least mean-square classification error is reduced. The error is expressed as [9]

∑

=

− −

= n

i

ij i j j T ij i j j

j d y x w d y x w

E

1

)) , ( (

2 1

, (1)

where d_jdenotes the desired output of node j

corresponding to inputx_i, n is the number of training

patterns and y_j(x_i,w_ij) denotes the vector output of the networks corresponding to input x_i and weight matrix

{ }

wij

W = . During the association or classification phase, the trained neural network itself operates in a feed-forward manner. The error is therefore a function of the weights of the input and output layers.

The backpropagation algorithm is a gradient descent method minimizing the mean square error between the actual and target outputs of a multilayer perceptron. Using the sigmoid nonlinearity

net i

e net

f ₋

− =

1 1 )

( , (2)

the backpropagation algorithm consists of the following steps: First, initialize all weights and node offsets to small random values. Second, present continuous input vector x_i and

specify desired output d_j. The output vector elements are set to zero values except for that corresponding to the class of the current input. Third, calculate the actual output vector y using the sigmoid nonlinearity. Fourth, adjust the weights by

i j ij

ij t w t x

w ( +1)= ()+ηδ , (3)

(3)

This program is used to train a set of prototypes from the recorded data. Each prototype corresponds to one particular stimulus. The classifier will then use these prototypes in the (offline) classification of the EEG signals. In order to train a new set of prototypes, the processed data of one recording is loaded into the program. The stimuli are labeled in the data. The data is then divided into training and validation sets in such a way that the three first sessions go to the training set and the one remaining session goes to the validation set. The thresholds affect how easily the signals (data sets) are classified as belonging to one of the stimuli and how easily they are rejected. The classifier computes probability values for a signal belonging to each of the stimuli included in the dataset. Then the highest probability value is chosen, and this value is compared to the probability threshold. If the value exceeds the threshold, the sample is classified to the corresponding stimuli, otherwise it is rejected. After the number of the iterations and the thresholds are adjusted, the training of the new prototypes can begin. During the training, a new set of prototypes are trained in every iteration. After the training has finished, the weight matrices corresponding to each iteration are reviewed. The best prototypes can then be saved to be used later in the validation set.

IV. RESULTSANDDISCUSSION

Preparatory to an analysis of the features of maximum amplitude from EEG-SSVEP signals, actual signals were recorded in a five-channels configuration. The raw data (Fig. 2, subject 1) were first pre-processed using a sixth-order band-pass filter (BPF) with cut-off frequencies of 4 Hz and 30 Hz where have the more significant signals than the other bands, see Fig. 3. These cut-off frequencies were chosen according to

the EEG-SSVEP frequency about 3 to 51 Hz. The amplitude of filtered signals was reduced from 100 to about 40 micro volt. Fig. 4 indicated the position of captured signal for feature extraction of each stimuli which are about 10s at the end of each clip movie. This position has been tested which provide maximum amplitude by means each subject reach the peak emotion of the movie. Fig. 5 is the filtered signals for 10s captured of calm emotion signals. Each 10s captured signal were extracted using FFT algorithm to obtain the frequency and the maximum amplitude as an emotion feature. The extracted feature is given in Fig. 6.

In the present study, a backpropagation neural networks classifier was used to classify the extracted emotion signals. In order to cope with nonlinearly separable problems, additional layers of neurons placed between the input layer and the output neuron are needed, leading to the multilayer perceptron architecture. At the outset, the structure of the network is chosen, after which the validation pattern appears in the graph window, and the network initialization values are introduced. Each subsequent layer has a weight coming from the previous layer. Performance is measured according to the specified performance function such as mean square error. The extracted signals from six channels of each emotion were feed into the neural networks with ten hidden layer to obtain one output networks as an emotion (sad, angry, happy, of calm). Table 1 indicates the extracted amplitude from each channels of each subject for sad, angry, happy, and calm emotions, and the accuracy of each subject. From Table 1, the 100% accuracy were obtained by subjects 1 to 3; the 75% accuracy was obtained by subject 7 (angry, happy, and calm, respectively); the 50% accuracy were obtained by subjects 5, 6, and 8 (happy and calm, respectively); the 25% accuracy was obtained by subject 4 (happy). Five subjects were fail to identify the sad emotion, three subject were fail to identify sad and angry emotions, and one subject was fail to identify sad, angry, and happy emotions. Each subject successfully identifies the calm emotion. The failure of several subject to identify some emotions could be caused by the limitation of the extractor and the classifier algorithms or the differences in emotional level of each subject in animating the stimuli movie. Another possibility also could be caused by the quality of the stimuli elections, by mean the short movie was not enough time to arouse the movie emotions.

(4)

Fig. 3. Filtered EEG-SSVEP signals using BPF

Fig. 4. Captured position of EEG-SSVEP signals used for emotion feature extracton

Fig. 5. 10s captured EEG-SSVEP signals used for emotion (calm) feature extracton.

TABLE 1.

EXTRACTED AMPLITUDE FROM EACH CHANNELS OF EACH SUBJECT FOR SAD,

ANGRY, HAPPY, AND CALM EMOTIONS, AND THE ACCURACY OF EACH SUBJECT. S Channel Sad Angry Happy Calm Accuracy

(%)

S

u

b

je

ct 1

Ch. F7 4,62 6,19 6,47 7,32

100 Ch. T5 4,70 6,26 6,86 7,47 Ch. O1 3,89 5,11 5,78 6,19 Ch. Fz 5,42 7,32 7,75 8,15 Ch. O2 2,28 2,08 2,76 4,02

S

u

b

je

ct 2

Ch. F7 10,71 9,34 7,56 9,43

100 Ch. T5 10,73 9,38 8,82 9,80 Ch. O1 10,25 9,84 8,29 11,15 Ch. Fz 9,19 8,22 6,30 8,29 Ch. O2 3,61 5,96 6,00 6,59

S

u

b

je

ct 3

Ch. F7 4,88 6,79 6,69 3,43

100 Ch. T5 3,39 4,76 4,88 2,72 Ch. O1 4,15 5,58 5,65 3,36 Ch. Fz 10,96 15,50 15,42 7,66 Ch. O2 4,47 6,36 6,93 3,84

S

u

b

je

ct 4

Ch. F7 3,91 3,74 3,27 4,86

25 Ch. T5 4,66 3,59 3,44 4,76 Ch. O1 4,93 3,97 3,37 4,82 Ch. Fz 3,36 2,72 3,00 3,72

Ch. O2 2,53 3,25 2,51 2,81

S

u

b

je

ct 5

Ch. F7 2,09 3,59 5,08 2,75

75 Ch. T5 1,21 0,92 1,38 1,14 Ch. O1 2,02 2,27 3,34 2,12 Ch. Fz 1,82 3,82 5,37 2,795 Ch. O2 1,79 2,99 3,43 1,72

S

u

b

je

ct 6

Ch. F7 4,01 5,16 7,19 3,29

50 Ch. T5 4,01 4,77 7,41 3,00 Ch. O1 4,29 5,58 7,29 3,32 Ch. Fz 3,10 4,26 6,16 2,62 Ch. O2 1,92 2,03 2,79 2,46

S

u

b

je

ct 7

Ch. F7 3,71 2,45 2,24 1,86

75 Ch. T5 1,61 1,34 2,32 1,30 Ch. O1 2,52 1,94 3,02 2,01 Ch. Fz 4,27 3,10 2,66 2,47 Ch. O2 2,90 2,62 2,48 1,51

S

u

b

je

ct 8

Ch. F7 2,43 1,32 3,41 2,44

50 Ch. T5 1,21 1,70 3,02 1,35 Ch. O1 1,83 2,40 3,48 1,84 Ch. Fz 2,88 2,54 4,03 2,87 Ch. O2 3,19 3,18 5,67 3,97

Fig. 6. Calm Extracted EEG-SSVEP signals: Amplitude and frequency

V. CONCLUSION

The results presented in this study show that an EEG-SSVEP signals can be used as a BCI system to communicate an emotion of the user or subject. The average of 75% classification accuracy is achieved for sad, angry, happy, and calm emotions. This result still highly possible to be enhanced by improve the experiment paradigm and signal processing algorithm. Our analysis may be later expanded for the construction of an automated tool for the understanding of children or disable subject.

ACKNOWLEDGMENT

This research was supported by the thematic program through the Bandung Technical Management Unit for Instrumentation Development (Deputy for Scientific Services) and the flagship program through the Research Center for Physics (Deputy for Engineering Sciences) funded by Indonesian Institute of Sciences, Indonesia.

REFERENCES

[1] F. Beverina, G. Palmas, S. Silvoni, F. Piccione, and S. Giove, "User adaptive BCIs: SSVEP and P300 based interfaces," PsychNol. J. vol. 1, pp. 331–54, 2003.

(5)

on the steadystate visual evoked potential in the 20-28 hz range,” Cognitive BrainResearch, vol. 6, pp. 249-26, 1997.

[4] R. Singla, A. Khosla, and R. Jha, “Influence of stimuli colour in SSVEP-based BCI wheelchair control using support vector machines,” J Med. Eng. Technol., vol. 38(3), pp. 125-34, Feb. 2014.

[5] T. Kaufmann, A. Herweg, and A. Kübler, “Toward brain-computer interface based wheelchair control utilizing tactually-evoked event-related potentials,” Journal of NeuroEngineering and Rehabilitation, vol. 11:7, 2014

[6] D. Huang, D. Y. Fei, W. Jia, X. Chen, and O. Bai, “Electroencephalography (EEG)-based brain-computer interface (BCI): a 2-D virtual wheelchair control based on event-related desynchronization/synchronization and state control,” IEEE Trans Neural Syst Rehabil Eng., vol. 20(3), pp. 379-88, 2012.

[7] A. Turnip, K. S. Hong, and M. Y. Jeong, “Real-time feature extraction of P300 component using adaptive nonlinear principal component analysis,” BioMedical Engineering OnLine,” vol. 10(83), 2011. [8] A. Turnip and K. S. Hong, “Classifying mental activities from

EEG-P300 signals using adaptive neural network,” Int. J. Innov. Comp. Inf. Control, vol. 8(7), 2012.

[9] A. Turnip, S. S. Hutagalung, J. Pardede, and, D. Soetraprawata, "P300 detection using multilayer neural networks based adaptive feature extraction method", International Journal of Brain and Cognitive Sciences, vol. 2, no. 5, pp. 63-75, 2013.

[10] A. Turnip and M. Siahaan, “Adaptive Principal Component Analysis based Recursive Least Squares for Artifact Removal of EEG Signals,” Advanced Science Letters, vol. 20, no.10-12, pp. 2034-2037(4), October 2014.

[11] A. Turnip and D. E. Kusumandari, “Improvement of BCI performance through nonlinear independent component analisis extraction,” Journal of Computer, vol. 9, no. 3, pp. 688-695, March 2014.

[12] A. Turnip, D. Soetraprawata, and D. E. Kusumandari, “A Comparison of Extraction Techniques for the rapid EEG-P300 Signals,” Advanced Science Letters, vol. 20, no. 1, pp. 80-85(6), January, 2014.

[13] A. Turnip, D. Soetraprawata, D. E. Kusumandari, “A comparison of EEG processing methods to improve the performance of BCI,” International Journal of Signal Processing Systems, 1 (1), 63-67 [14] A. Turnip and D. Soetraprawata, “The Performance of EEG-P300

Classification using Backpropagation Neural Networks,” Journal of Mechatronics, Electrical Power, and Vehicular Technology, 4 (2), 81-88 [15] A. Turnip and D. Soetraprawata, “Electrooculography Detection from

Recorded Electroencephalogram Signals by Extended Independent Component Analysis,” Advanced Science Letters, 21 (2), 173-176, 2015.

[16] A. Turnip, “Comparison of ICA-Based JADE and SOBI Methods EOG Artifacts Removal”, Journal of Medical and Bioengineering, Vol 4 (6), 2015.

[17] A. Belitski, J. Farquhar, and P. Desain, “P300 audio-visual speller,” Journal of Neural Engineering, vol. 8(2), 025022, 2011.

[18] J. P. Rosenfeld, B. Cantwell, V. T. Nasman, V. Wojdac, S. Ivanov, and L. Mazzeri, “A modified, event-related potential-based guilty knowledge test,” Int. J. Neurosci., 42(1-2), 157-161, 1988.

[19] M. Kh. Hazrati and A. Erfanian, “An online EEG-based brain-computer interface for controlling hand grasp using an adaptive probabilistic neural network,” Medical Engineering & Physics, vol. 32(7), pp. 730-739, 2010.

[20] J.M. Zurada, Introduction to Arti"cial Neural Systems West Publishing Company, St. Paul, MI, 1992.

[21] P. Sajda, A. Gerson, R. Müller, B. Blankertz, and L. Parra, “A Data Analysis Competition to Evaluate Machine Learning Algorithms for Use in Brain-Computer Interfaces,” Computer Journal of IEEE Trans Neural Systehms, vol. 11, no. 2, pp. 184-185, 2003.

Arjon Turnip received the B.Eng. and M.Eng. degrees in Engineering Physics from the Institute of Technology Bandung (ITB), Indonesia, in 1998 and 2003, respectively, and the Ph.D. degree in Mechanical Engineering from Pusan National University, Busan, Korea, under the World Class University program in 2012. He is currently work in the Technical Implementation Unit for Instrumentation Development, Indonesian Institute of Sciences, Indonesia as a research coordinator. He received Student Travel Grand Award for the

best paper from ICROS-SICE International Joint Conference 2009, Certificate of commendation: Superior performance in research and active participation for BK21 program from Korean government 2010, JMST Contribution Award for most citations of JMST papers 2011, Inventor Technology Award from Minister of RISTEKDIKTI 2015, and Bupati Samosir Award for the role and activities of Samosir Development. His research areas are integrated vehicle control, adaptive control, nonlinear systems theory, estimation theory, signal processing, brain engineering such as brain-computer interface. Dr. Arjon is an IEEE Member. He is an Editor of the Widyariset Journal and the Journal of Mechatronics, Electrical Power, and Vehicular Technology (JMEV). He is the Guest Editor at the Indonesian Internetworking Journal (IIJ). He was the General Chair of the 2nd 2014 International Conference on Technology, Informatics, Management, Engineering and Environment (TIME-E), the 2015 International Conference on Industrial Internet of Things (ICIIoT), the 2015 International Conference on Automation, Cognitive Science, Optics, Micro Electro--Mechanical System, and Information Technology (ICACOMIT), Co-General Chair of the 3rd 2015 International Conference on Technology, Informatics, Management, Engineering and Environment (TIME-E), and General Chair of the 2016 International Conference on Informatics, Robotics, Network, Control and Systems (IRONCONS 2016). He is a reviewer of the Journal of Neuroscience Methods, Biomedical Signal Processing and Control, and Advances in Mechanical Engineering. Meanwhile, he accomplished four governmental projects as a project leader entitled the development and application of assistive intelligent system based bio-feedback signal for clinic instrumentation, suspension system development, implementation of water pump system based renewable hybrid energy (Funding support by Indonesian Institute of sciences). The first three projects are collaborated with Medical School (Padjajaran Univ.)/Hasan Sadikin Hospital, Astra Otopart Company, and the local government of the Kabupaten Samosir, respectively.

M. Faizal Amri received the B.Sc degree in Electronic and Instrumentation from Gadjah Mada University (UGM). He is currently work in the Technical Implementation Unit for Instrumentation Development, Indonesian Institute of Sciences, Indonesia as a junior researcher. His research interest are in electronics and control system, Artificial Inteligent, signal processing, and brain-computer interface.

. Artha Ivonita Simbolon B.Sc degree in Physic from Intitute Technology of Bandung (ITB). She is currently work in the Technical Implementation Unit for Instrumentation Development, Indonesian Institute of Sciences, Indonesia as a junior researcher. Her research interest are in electronics and control system, Artificial Inteligent, signal processing, and brain-computer interface.

Poltak Sihombing (PS) was born in Indonesia on March 17, 1962. He received the Physics degree in Computation Physics from the University of Sumatera Utara, Medan, Indonesia, in 1988. The M.Sc. degree in Computer Science from the University of Indonesia (UI), Jakarta, Indonesia, in 1989, and the Computer Science Ph.D. degree in Intelligent System Techniques from the Universiti Sains Malaysia (USM), Penang, Malaysia, in 2010. He has held lecturing positions at The University of Sumatera Utara, Medan, Indonesia. Since 2011, He has been the head of Computer Science Department on Faculty of Computer Science and Information Technology, University of Sumatera Utara. His research interests cover the intelligent system techniques, instrumentation engineering and information retrieval systems. He is a member of APTIKOM (Assosiasi Perguruan Tinggi Informatika Komputer/ The Association of Universities Computer and Informatics). Since 2011, he served as a secretary of North Sumatera and Aceh region.