RTIFICIAL N EURAL N ETWORKS P ARAMETERS AND A E MOTION R ECOGNITION USING V OCAL T RACT

(1)

International Journal of Engineering Science Invention Research & Development; Vol. III, Issue I, July 2016 www.ijesird.com, e-ISSN: 2349-6185

Surabhi Vaishnav and Saurabh Mitra ijesird, Vol. III, Issue I July 2016/54

E MOTION R ECOGNITION USING V OCAL T RACT

P ARAMETERS AND

A RTIFICIAL N EURAL N ETWORKS

1Surabhi Vaishnav, ²Saurabh Mitra

1M.Tech Scholar, ²Asst. Professor

Department of Electronics & Communications Engineering Dr. C. V. Raman University

1[email protected],

2[email protected] ABSTRACT- The development of an automated system with less number of speech parameters and an ultimate classifier for emotion recognition has become an importance in this era. The reason behind this is that the human-machine interface is being used in various fields like medical, engineering, cognitive solutions, etc. The paper describes the ways to extract speech parameters for emotion recognition and the emotion classification procedure using the features. The Mel-Frequency Cepstral Coefficients, delta Mel-Frequency Cepstral Coefficients, delta-delta Mel-Frequency Cepstral Coefficients, pitch and formant frequencies are used as speech parameters for the emotion detection and ultimate feature combination is defined from various speech feature combinations and accuracy obtained from classification. The artificial neural network is used as classifier to distinguish the emotions.

General Terms

Emotion, ANN, vocal tract features

Keywords

Voice activity detection, MFCC, delta MFCC, delta-delta MFCC, pitch, formant frequency, accuracy, confusion matrix, sensitivity, specificity

1. INTRODUCTION

Human-machine interaction (HMI) has brought a great importance in recent years. For an effective and natural HMI, emotion recognition plays a vital role. Emotions reflect the mental state of the person through speech, facial expressions, body postures and gestures and also other physical parameters like body temperature, blood pressure, muscle action, etc. The mental state of the person indirectly affects the speech produced by the person. E.g. in human- human interaction, speech rate is faster in case of anger/ joy and pitch range is also wider while in case of sadness, speech is slower with lower pitch range. Therefore, emotion detection in speech is advantageous in various applications.

There are two fundamental processes performed in speech processing i.e. signal modeling and pattern recognition. The speech signal is converted

into a set of parameters using signal modeling while, pattern recognition involves searching for parameter sets from memory that closely matches the feature set obtained from unknown speech input signal. The speech understanding system would require tremendous advances in linguistics, natural language, and knowledge of everyday human experience and widely used in several applications, especially telecommunication systems.

The aim of the work stated in this paper is to design a robust system for speech emotion recognition using minimum number of speech features, defining an ultimate set of parameters for emotion recognition and formulating artificial neural network’s back-propagation model.

Researchers have contributed in features extraction automation and have used various available machine learning techniques like Artificial Neural Networks, Support Vector Machines, Hidden Markov Model, Gaussian Mixture Model, etc.

Emotions are involved when one communicates with others. Emotions can be categorized as neutral, sad, anger, happy, boredom, disgust. Emotions add ornaments to the conversation. The existing methods for emotion detection from voice use mainly MFCC and energy features. Efforts are being made to have a robust system for the detection purpose, which will be cost effective, consisting minimum number of parameters and time saving. A lot of work has been done in the area of speech emotion recognition. Some of the researches were carried out on emotion detection by using acoustic analysis of speech. Tin Lay Nw et.al proposed a text independent method of speech emotion classification, which makes use of short time log frequency power coefficients (LFPC) to

(2)

Surabhi Vaishnav and Saurabh Mitra ijesird, Vol. III, Issue I July 2016/55 represent the speech signals and a Hidden Markov

Model (HMM) is used as a classifier [6]. The six categories of speech emotion i.e. anger, disgust, joy, fear, sadness and surprise were used in this work.

The average accuracy achieved using this system is 78% [6]. Also, the performance analysis was done using mel-frequency cepstral coefficients (MFCC) and linear prediction cepstral coefficients (LPCC).

Kamran Soltani et.al studied the importance of the psychology and the linguistics in spoken language man-machine interfaces. Along with the techniques in signal processing and analysis, it also requires psychological and linguistic analysis. The work makes use of six emotions, happiness, sadness, anger, fear, neutral and boredom. It uses pitch i.e. speech fundamental frequency, formant frequencies, energy and voicing rate as features [12]. These speech parameters were used to train neural network classifier and the Berlin Database of Emotional Speech is used in this work. The average accuracy attained using this system is around 77%

[12]. The work concludes that anger and neutral can be recognized easily while fear the most difficult one. Energy and pitch are the two important features for speech emotion recognition according to this paper. Jana Tuckova et.al performed experimental analysis using speech parameters like fundamental frequency, formant frequency and statistical analysis was conducted for multi-layer neural network. The average accuracy obtained using this technique is 75.93% for multiword sentences while that for one word sentences is 81.67% [15]. Mandar Gilke performed MFCC based vocal emotion recognition using ANN in which MFCC features were used as speech parameters and 5 emotion states viz. neutral, happy, sad, anger and surprise were considered for analysis. The average accuracy of recognition achieved is 60.55% [3].

A system was proposed that allows recognition of a person?s emotional state by Igor Bisio et.al.

The system provides an effective solution to improve human-computer interaction allowing human and computer intelligent interaction. The system built is able to recognize six emotion states viz. anger, boredom, happiness, sadness, disgust,

fear and the neutral state. The system does two experiments i.e. gender recognition and emotion recognition. Together with MFCC, pitch is the most frequently used parameter in recognizing speaker?s gender[1]. Other speech parameters used are formants, bandwidths, source spectral tilt, jitter and shimmer, harmonic to noise ratio. Speech features used for emotion recognition are statistical analyses of amplitude of speech, energy, pitch, formants, 12 MFCC, pitch and amplitude perturbations [1].

Berlin Emotion Speech database is employed in this research work. Support vector machine (SVM) classifier is used for the purpose of emotion recognition in this work.

Mohit Shrivastava made use of pitch, formant frequency, energy, MFCC as speech parameters on Berlin Emotion Speech database and classified the emotions (anger, happiness, sadness, fear and disgust) using artificial neural networks (ANN) and SVM where SVM was found to perform better than ANN[13].

Jouni Pohjalainen et.al investigated a filtering technique in automatic detection of emotions from telephonic speech where the MFCC, delta MFCC and delta-delta MFCC features were incorporated with Gaussian mixture model (GMM) as a classifier for emotion classification on Berlin database of emotional speech, while autoregressive (AR) model is employed in the proposed filtering method[7].

The work presented in this paper contributes in emotion recognition using features like MFCC, delta MFCC, delta-delta MFCC, pitch, formant frequencies. The system performance was evaluated using different feature combinations and ANN classifier using measures like confusion matrix, sensitivity, specificity and accuracy. The work makes use of three emotions namely happy, neutral and sad.

The remaining paper is ordered as follows: section II explains proposed work, section III gives information about the database, section IV explores classifier used while, section V explains the results obtained.

(3)

Surabhi Vaishnav and Saurabh Mitra ijesird, Vol. III, Issue I July 2016/56 2. PROPOSED WORK

The aim behind the work is to formulate a system using acoustic analysis and parametric and non- parametric features for the distinction between different emotions. The block diagram of the algorithm is shown as in fig. 1. The algorithm proceeds as preprocessing, feature extraction, classifier training and decision making. Various feature combinations are then tested using designed classifier and an ultimate set of features is formed.

Then unknown test speech recording is given as an input to the trained model to classify into the emotion it corresponds.

Fig. 1. System Block Diagram

2.1 PREPROCESSING

This is the important step in any speech processing application. It involves normalization, framing, windowing, voice activity detection.

2.1.1 Normalization.

Zero mean and unity variance normalization is applied to all the speech recordings of three speech emotion directories to remove any dc offset present.

2.1.2 Framing, Windowing and Overlapping.

The movement of the glottal muscles cannot achieve infinity rate and is quasistationary for for about 10 ms. The speech signal properties are assumed to be stable for short time 10-30 ms[8].

Therefore the whole signal is split into 20 ms frames.

The long length signal is converted into finite one using windowing [11]. The work makes use of hamming window. The window function for hamming window is given as in equation 3.1:

(1)

At the ends of the window, de-emphasis occurs.

To avoid this, overlapping of frames i.e., the new frame consists of some part of former frame and the forthcoming frame, is done [5]. The overlap of 70%

has been chosen in this work.

2.1.3 Voice Activity Detection (VAD).

In case of speech emotion recognition, detection of speech needs to be corrected which reflects the resulting recognition accuracy. VAD is needed to detect start point and end point correctly [10]. In this work, VAD is carried out depending on short time energy and zero crossing rate. VAD calculates the length of the signal and defines the number of segments. It also calculates zero crossing rate (ZCR) and short time energy (EN) and compares the result with the threshold. By making the use of this, finds out starting and end point of the signal and at last, we get the only voiced part of the speech signal [10].

2.2 FEATURE EXTRACTION

The work uses MFCC, delta MFCC, delta-delta MFCC, pitch and formant frequencies as speech parameters.

2.2.1 Mel-Frequency Cepstral Coefficients (MFCC).

The actual frequency (f) is measured in Hertz (Hz) and a subjective pitch is measured on mel scale [14]. The mel scale relates the perceived frequency of a pure tone to its actual measured frequency. The lower frequencies are differentiated better than high frequencies by humans. The mel scale is a linear frequency spacing below 1000 Hz and a logarithmic spacing above 1000 Hz. As a reference point the pitch of a 1 kHz tone, 40 db above the perceptual hearing threshold is defined as 1000 mels. So, we can use the following formula to compute the mels for a given frequency f in Hz,

(4)

Surabhi Vaishnav and Saurabh Mitra ijesird, Vol. III, Issue I July 2016/57 mel(F)=2595∗ log10(1+ f/700) (2)

The mel-filters can be represented as shown in fig 2.

The steps to calculate MFCC are shown in fig. 3 below: The first step is pre- Filtering of the speech signal.

Fig. 2. Mel-filter Bank

Fig. 3. Calculation of MFCC

Then the speech signal is framed and windowed. As mentioned earlier, the work makes use of Hamming window. 13 MFCC coefficients are extracted for each frame. Then apply Fast Fourier Transform (FFT) to the windowed signal which will give the product of excitation component and vocal tract response. Then compute the mel-filter bank which in this work is a set of 24 triangular filters which we apply to the transformed signal obtained and we obtain a mel-scale as explained earlier in this section[14]. In next step, we convert the log mel-spectrum back to time. The result is called the mel-frequency ceptsrum coefficients (MFFC). This representation provides a good representation of the local spectral properties of the signal for the given frame analysis because the MFCC are real numbers [13]. We can convert them to time domain using Discrete Cosine Transform (DCT). The resulting features are called as MFCC coefficients.

2.2.2 Delta and Delta Delta MFCC.

These are also known as differential and acceleration coefficients. The power spectral envelope of a single frame is described by MFCC parameter, but the speech may contain more information in the form of dynamics i.e. MFCC trajectories. It has been found that appending these with MFCC has proved the performance of the recognition system. Deltas are just the differentiation of MFCCs while Delta Delta are differentiation of Deltas.

2.2.3 Formant Frequencies.

These are the resonances of the vocal tract depending on the shape of it. The work makes use of Linear Prediction Coefficients (LPC) to calculate formants from the coefficients, then evaluate the poles p which represent the vocal tract model. The spectrum of the vocal tract will give the formant frequencies. The paper considers three formant frequencies. LPC analysis is the approximation of the speech sample as a linear combination of past speech samples. A discrete set of prediction coefficients is obtained by minimizing the sum of the squared differences between the linearly predicted speech samples and the actual ones [8].

The autocorrelation method has been used to find out LPC.

2.2.4 Pitch.

The intrinsic and extrinsic muscles of the larynx regulate frequency of vibration of vocal folds [9]. The vocal folds vibrate at different rates because of the in-built tension in vocal cords and the sub-glottal air pressure forms the pitch frequencies [11]. The frequency of the pulses emitted by the opening of the vocal cords determines the pitch of the larynx tone. The change in frequency means a change in the time occupied by one cycle, i.e., the closed phase plus the open phase [9]. The pitch is the fundamental frequency (F0) of vibration which is related to the periodicity and represents the rate of vibration of vocal folds during voiced sounds. Average value of F0 is 125 Hz in men, 225 Hz in women and 265 Hz in children [6]. The range of F0 is 60 Hz to 500 Hz [9]. The autocorrelation method to determine pitch is the most convenient way to represent the pitch as

(5)

Surabhi Vaishnav and Saurabh Mitra ijesird, Vol. III, Issue I July 2016/58 a function of time. The autocorrelation function

reaches maximum value at 0, ± K, ± 2K, ..., where, K is the period of the signal. PRAAT is used to find out pitch and variations in pitch, which is a well- known software for voice analyses in speech processing area and is developed by scientists working in the area of speech processing. It has been observed that the pitch varies from 169 to 341 Hz for happy emotion, for sad, it varies from 89 to 164 Hz while, for neutral emotion, variation is from 104 to 190 Hz.

3. SPEECH DATABASE

The mix-gender Berlin Speech Emotional Database is used in this work, consisting of total 209 speech recordings out of which 70 correspond to emotion happy, 78 speech samples are from emotion neutral while emotion sad has 61 recordings. The utterances were recorded from 10 different actors and actresses with 10 different texts in various emotions i.e. happy, boredom, neutral, sad, angry, disgust and anxiety. As the study is focused on only three emotions viz. happy, neutral and sad, it makes use of only corresponding recordings. The factors considered for speech recording are mentioned as age, gender, distance of microphone from speaker’s mouth, sampling rate and bit resolution used for each recording, recording length and software used for recording [2].

4. ARTIFICIAL NEURAL NETWORKS The pattern matching is performed using artificial neural networks (ANN). Specifically back- propagation neural network is used. The classifier used is shown as in fig. 4. There is an interconnection of neurons and each interconnection has weights associated with it [17].

The model uses learning delta rule which updates the weights and minimizes the mean squared error between actual output values and the desired ones.

The weights are updated until the error calculated becomes tolerable [17]. The model is also trained number of times using different hidden layers and hidden neurons to get greater accuracy.

Fig. 4. ANN model

Different feature combinations are designed for the emotions considered and a target matrix is designed representing the emotions. The feature set and target set are used to train the network designed. Once the ultimate gradient is reached and desired confusion matrix is obtained, an unknown speech sample is given to the trained classifier. It then performs pattern matching and classifies the speech into emotion happy, neutral or sad.

5. EXPERIMENTS AND RESULTS The section describes the accuracy obtained using different combinations of features and it also gives the optimum set of features for the emotion recognition from speech. The work makes use of pitch, formant frequencies, MFCC, dMFCC and ddMFCC features while the combinations used are (MFCC, pitch, dMFCC, ddMFCC), (MFCC, dMFCC, ddMFCC, formants) and (MFCC, dMFCC, ddMFCC). The ANN-BPNN classifier is used to classify the emotions Hppy, Neutral and Sad using the feature combinations mentioned earlier in the section. The performance of the system is evaluated using various measures like confusion matrix, while the terms used in the confusion matrix are true positive (TP), true negative (TN), false negative (FN) and false positive (FP) which are defined later in the section.

The terms used in the confusion matrix can briefly be described and are as shown in table 1. The terms true negative (TN), true positive (TP), false positive (FP) and false negative (FN) are defined as follows:

1) True positive (TP): True event detected as true [16]. 2) True negative (TN): False system is classified as false. It is the sum of all columns and

(6)

Surabhi Vaishnav and Saurabh Mitra ijesird, Vol. III, Issue I July 2016/59 rows excluding that class’s column and row [16]. 3)

False negative (FN): True event detected as false. It is the sum of corresponding row excluding TP [16].

4) False positive (FP): False event discriminated as true. It is the sum of corresponding column excluding TP [16]. 5) ENH: The speech recordings corresponding to class happy are classified as neutral. 6) E_HS: The speech recordings corresponding to class happy are classified as sad.

7) ENH: The speech recordings corresponding to class neutral are detected as neutral. 8) ENS: The speech recordings corresponding to class neutral are detected as sad. 9) ESH: The speech recordings of class sad are discriminated as happy.

Table 1. Understanding TP, TN, FP, FN CLASSIFICATION

PREDICTION

HAPPY NEUTRAL SAD

HAPPY TP_H EHN EHS

NEUTRAL ENH TP_N ENS

SAD ESH ESN TP_S

10) ESN: The speech recordings of class sad are classified as neutral. 11) Sensitivity: The probability of classifying the event as true when it is true, is termed as sensitivity (SE)[4].

(3)

12) Specificity: The probability of classifying the event as false provided it is false is called as specificity (SP)[4].

(4)

13) Accuracy (AC): The overall performance of the system is defined with a parameter accuracy which can be stated form the probability on correct classification [4], which is the ratio of sum of correct classifications and total number of classifications.

(5) From the experiments carried out using different combinations of features considered i.e. (MFCC, delta MFCC, delta delta MFCC), (MFCC, delta MFCC, delta delta MFCC, formants) and (MFCC, pitch, delta MFCC, delta delta MFCC), the first combination of features gives the best results in terms of accuracy, sensitivity and specificity. This states that MFCC parameter plays an important role

in case of emotion recognition. This section relates the overall accuracy obtained for feature combinations considered and it also relates the sensitivity and specificity obtained emotion–wise, as it is different for different emotion considered.

From the experiments carried out for various feature combinations, it has been found that the accuracy for feature combination, (MFCC, DMFCC, D2MFCC), is best than any other combination which is illustrated in table 2 below.

Table 2. Accuracy Obtained for Different Feature Combinations

FEATURES ACCURACY

MFCC, DMFCC, D2MFCC 78.46%

MFCC, PITCH, DMFCC, D2MFCC 75.59%

MFCC, DMFCC, D2MFCC, FORMANTS

78%

The classifier performance for the combinations of features considered is given in successive tables 3, 4, 5. The sensitivity and specificity for each emotion for corresponding feature combination is calculated from the formulae stated earlier in this section.

Table 3. Classifier performance for Feature Combination MFCC, DMFCC, D2MFCC

HAPPY NEUTRAL SAD

SENSITIVITY 70 82.05 83.60

SPECIFICITY 100 78.62 88.51

Table 4. Classifier performance for Feature Combination MFCC, PITCH, DMFCC, D2MFCC

HAPPY NEUTRAL SAD SENSITIVITY 65.71 80.76 80.32 SPECIFICITY 100 80.32 87.83

Table 5. Classifier performance for Feature Combination MFCC, DMFCC, D2MFCC, FORMANTS

HAPPY NEUTRAL SAD SENSITIVITY 71.42 83.33 78.68 SPECIFICITY 100 77.09 89.18

From tables 3, 4 and 5, it can be seen that the performance measures obtained for the combination (MFCC, DMFCC, D2MFCC) are more prominent than rest feature combinations as the true positives,

(7)

Surabhi Vaishnav and Saurabh Mitra ijesird, Vol. III, Issue I July 2016/60 true negatives values are also prominent for this

feature combination than other combinations. The confusion matrix for the combination giving the best accuracy is illustrated as in table 5.6, which represents the total accuracy obtained i.e. 78.46%, specificity and sensitivity are also mentioned in the confusion matrix.

Table 6. Confusion matrix of feature combination MFCC, DMFCC, D2MFCC

CLASSIFICATION PREDICTION

HAPPY NEUTRAL SAD

HAPPY 49 18 3 70%

NEUTRAL 0 64 14 82.05%

SAD 0 10 51 83.60%

100% 69.56% 75% 78.46%

From the experimental illustrations on different feature combinations for emotions happy, neutral and sad, it could be concluded that the features MFCC, DMFCC, D2MFCC form the best feature combination among rest feature combinations considered.

6. CONCLUSION

The work analyses various feature combinations from features MFCC, DMFCC, pitch, D2MFCC, formants for emotion detection. This work highlights the recent developments and different approaches to provide technological perspective of the study. It is found that the system gives the best performance to (MFCC, DMFCC, D2MFCC) feature combination than rest of the combinations considered. This concludes that pitch and formants don’t contribute much in emotion recognition in order to improve the system performance to the great extent. Also, the confusion matrix for the optimum feature set is plotted which represents the total accuracy, specificity, sensitivity. This indicates that the MFCC features are the most powerful features in case of emotion recognition. A comparative study of classification of emotions happy, neutral and sad using feature combinations is carried out successfully.

7. ACKNOWLEDGMENT

The authors are very much thankful to all the professors and the Department of Electronics Engineering of Dr. C. V. Raman University. Authors also want to thank Director of Dr. C. V. Raman University.

REFERENCES

[1] Igor Bisio, Alessandro Delfino, Fabio Lavagetto, Mario Marchese, and Andrea Sciarrone. Gender-Driven Emotion Recognition Through Speech Signals for Ambient Intelligence Applications. 1(2), 2014.

[2] D.B. Fry. The Physics of Speech. Cambridge Textbooks in Linguistics. Cambridge University Press, 1979.

[3] Mandar Gilke, Pramod Kachare, Rohit Kothalikar, and Varun Pius Rodrigues. MFCC-based Vocal Emotion Recognition Using ANN. 49(Iceei):150–154, 2012.

[4] Juan Ignacio Godino-Llorente, Pedro Gomez-Vilda, and´

Manuel Blanco-Velasco. Dimensionality reduction of a pathological voice quality assessment system based on gaussian mixture models and short-term cepstral parameters.

IEEE Transactions on Biomedical Engineering, 53(10):1943–

1953, 2006.

[5] Ian McLoughlin. Applied Speech and Audio Processing with MATLAB Examples. Cambridge University Press, 2009.

[6] T Nwe. Speech emotion recognition using hidden Markov models. Speech Communication, 41(4):603–623, 2003.

[7] Jouni Pohjalainen and Paavo Alku. Multi-Scale Modulation Filtering in Automatic Detection of Emotions in Telephone Speech. Conference, Ieee International Processing, Signal, pages 6326–6330, 2014.

[8] L. Rabiner and R. Schafer. Digital Processing of Speech Signals. Englewood Cliffs: Prentice Hall, 1978.

[9] Lawrence Rabiner and Biing-Hwang Juang. Rabiner & Juang - Fundamentals of Speech Recognition.pdf. PTR Prentice Hall, 1993.

[10] J Ram´ırez, J M Gorriz, and J C Segura. Voice Activity Detec-

´ tion . Fundamentals and Speech Recognition System Robustness. (June):1–23, 2007.

[11] V Sellam and J Jagadeesan. Classification of Normal and Pathological Voice Using SVM and RBFNN. Journal of Signal and Information Processing, 05(01):1–7, 2014.

[12] Kamran Soltani and Raja Noor Ainon. Speech emotion detection based on neural networks. pages 7–9, 2007.

[13] M Srivastava and A Agarwal. Classification of emotions from speech using implicit features. ... and Information Systems (ICIIS), ..., 2014.

[14] Li Tan and Montri Karnjanadecha. Modified Mel-Frequency Cepstral Coefficient. Proceedings of the Information Engineering Postgraduate Workshop, pages 127–130, 2003.

[15] Jana Tuckova and Martin Sramka. Networks. (102):141–147, 2010.

[16] A. Visave, P. Kachare, A. Jeyakumar, A. Cheeran, and J.

Nirmal. Glottal pathology discrimination using ann and svm.

In Advances in Computing, Communications and Informatics

(8)

Surabhi Vaishnav and Saurabh Mitra ijesird, Vol. III, Issue I July 2016/61

(ICACCI), 2015 International Conference on, pages 1377–

1381, Aug 2015.

[17] Ashwini Visave, Pramod Kachare, Amutha Jeyakumar, Alice Cheeran, and Gurmit Bachher. Article: Vocal features for glottal pathology detection using bpnn. International Journal of Computer Applications, 118(17):1–6, May 2015.