Study on Recognition of human moods using Neural Network

(1)

International Journal of Advanced Engineering Science and Technological Research (IJAESTR) ISSN: 2321-1202, www.aestjournal.org @2016 All rights reserved

1

Study on Recognition of human moods using Neural Network

Dr.Prashant V.Ingole¹, Dr.Naresh P. Jawarkar², Mr.Sandeep A.Awachar³

1

Department of EXTC,G.H.Raisoni College of Engg. & Management, Amravati, Maharashtra, India

2

Department of EXTC,Babasaheb Naik College of Engineering, Pusad, Dist.-Yavatmal, Maharashtra ,India

3

Department of Computer Sci. & Engg., College of Engineering & Technology, Akola, Maharashtra ,India [email protected]

Abstract- The Knowledge enrichment is accomplished only through literature search on any topic of interest. Same is true even in case of human moods recognition. But for that it is necessary to have a broad approach towards the origin and developments on moods or facial expression recognition in the recent past. The contribution of Ekman and Friesen gave an opportunity that led to accept this as the topic of interest.

Huge number of contributions are made by many researchers to recognize the six primary expressions viz.

happiness, sadness, fear, disgust, surprise and anger. In this paper, we attempt to focus some of the important work on the subject.

Index Terms: mood, preprocessing, recognition, training

I. INTRODUCTION

Facial expressions are the means to convey moods, feelings, warning signs of dangers, happiness is appointment, confidence etc. of human. It is genetically injected into the living things from the womb to tomb. Psychologists, Saints and Men of spirituality consider facial expressions as indications of hidden truth and exposition of sudden feelings, in the right way, at the right time without any reservations. In man, facial expressions were well studied, since 1971 by the pioneers Ekman and Friesen [1]. Even in the theory of evolution of Darwin, there are reminiscence of the rule of automatic facial expression, to grab new shapes and intelligence in the transformation process of one animal into another. Ekman and Friesen are acclaimed of their contributions to the postulation of six primary facial expressions- happiness, sadness, fear, disgust, surprise and anger. These six distinctive expressions are unique in their feature [2].

A mood is an emotional state. Moods differ from emotions in that they are less specific, less intense, and less likely to be triggered by a particular stimulus or event. Moods generally have either a positive or negative valence. In other words, people typically speak of being in a good mood or a

bad mood.Fig.1 show the nine basic human moods.

Fig.1 .Nine basic human moods Mood also differs from temperament or personality traits which are even longer lasting. Nevertheless, personality traits such as optimism and neuroticism predispose certain types of moods. Long term disturbances of mood such as clinical depression and bipolar disorder are considered mood disorders. Mood is an internal, subjective state but it often can be inferred from posture and other behaviors. "We can be sent into a mood by an unexpected event, from the happiness of seeing an old friend to the anger of discovering betrayal by a partner. We may also just fall into a mood”. The mood list varies from : calm, cheerful, chipper, cold, complacent, confused, depressed, determined, devious, dirty, disappointed, discontent, energetic, enraged, enthralled, envious, frustrated, full geeky, giddy, giggly, happy, high irritated, jealous, jubilant, lazy, lethargic, sad, satisfied, shocked, sick, silly, sleepy, smart, stressed, surprised, touched, uncomfortable, weird, accepted, accomplished, aggravated, alone, amused, angry, and many more. These can be recognized from human expressions, using neural network classifier.

(2)

International Journal of Advanced Engineering Science and Technological Research (IJAESTR) ISSN: 2321-1202, www.aestjournal.org @2016 All rights reserved

2

II. LITERATURE REVIEW

In the early 1990’s the engineering community started to construct automatic methods of recognizing moods from facial expression in an image and videos, and many of computer studies that focused on moods recognition. Some studies use facial expressions to build a model for mood recognition, and others use another elements or factors to build the models like voice, pulse, body movements etc.

Abu Sayeed Md. Sohail, and Prabir Bhattacharya [3], used an automated facial expressions classification method that incorporates a hybrid image processing based facial feature point detection technique along with Support Vector Machines as the classifier. As shown by the experimental results, the method performs better when SVM is used as classifier rather than k-NN, Neural Network or Naive Bayes Classifier and provides average successful recognition rates of 89.44% and 84.86% respectively over two different facial expression databases.

Jen-Chun Lin, Chung-Hsien Wu and Wen-Li Wei [4], used an approach to the automatic recognition of human emotions from audio-visual bimodal signals using an error weighted semi-coupled hidden Markov model (EWSC-HMM).

The approach combines an SC-HMM with a state-based bimodal alignment strategy and a Bayesian classifier weighting scheme to obtain the optimal emotion recognition result based on audio-visual bimodal fusion. The state-based bimodal alignment strategy in SC-HMM is proposed to align the temporal relation between audio and visual streams. The Bayesian classifier weighting scheme is then adopted to explore the contributions of the SC-HMM-based classifiers for different audio-visual feature pairs in order to obtain the emotion recognition output. For performance evaluation, they considered two databases: the MHMC posed database and the SEMAINE naturalistic database. The results show that the approach not only outperforms other fusion-based bimodal emotion recognition methods for posed expressions but also provides satisfactory results for naturalistic expressions.

Matthew and Patterson [5] discuss a framework for the classification of emotional states, based on still images of the face, using active appearance model (AAM) and get distance using n Euclidean’s space as features. To train and test the classifier they chose to use the facial expression database known as “FEEDTUM” - Facial Expressions and Emotion Database , and seven basic emotions are used, happy, sad, angry, surprise, fear, disgust and natural state. The best results they obtained are in happy, natural and disgust emotions at the rate 93.3%, fear at 90.0%, and 79.7%. In surprise, angry and sad it is at rate 63.9%.

Raheja and Kumar [6] presented architecture for human gesture recognition, considering color image with different gestures by using back propagation Artificial Neural

Networks (ANN). Four stages applied in the approach, face detection, image reprocessing, training network and recognition model. The pre-processing stage contains three methods, histogram equalization, edge detection, thinning and token generation. The model was trained using the three different gesture images, happy, sad and thinking expressions of faces. The model was tested with 100 images of three gestures, the results were 94.28% for happy, 85.71% for sad and 83.33% for thinking.

Karthigayan et.al [7] used Genetic Algorithms (GAs) and Artificial Neural Networks to build human emotion classifier. This classifier detects six human emotions Neutral, Sad, Anger, Happy, Fear, Disgust (or Dislike) and Surprise.

They depend on two facial elements in the classifier, eyes and lips. By applying some preprocessing methods and edge detection, they extracted the eyes and lip regions, then extracted the features from these regions. Three feature extraction methods were applied: projection profile, contour profile and moments. The GA is applied to get the optimized values of the minor axes of an irregular ellipse corresponding to the lips and the minor axis of a regular ellipse related to eye by using a set of new fitness functions. Finally, they apply the results from GA on the ANNs model. Two architectures of Ann’s models are proposed with an average of 10 trials of testing. The achieved results of 3x20x7 and 3x20x3 of ANN architecture were 85.13% and 83.57% of success rate respectively. The successful classification even goes to the maximum of about 91.42% in the ANN model of 3x20x7 structure.

N. M. Hewahi, A.R. M. Baraka [8], tried to investigate the impact of ethnic group on accuracy of emotion recognition based on face expressions by proposing an emotion recognition approach using back propagation neural networks. They demonstrated that the ethnic group has a positive impact on the accuracy of identifying human emotions based on facial features. They got 75% accuracy regardless ethnicity, and 83.3% accuracy considering ethnicity.

A. Butalia, Dr. A.K. Ramani, Dr. Maya Ingle, Dr.

Parag Kulkarni [9], used facial expression of current image as a context to hand gesture which again acts as a context to the next image linking the images in a video to extract unstable sentiments. They have introduced the intensity levels of facial expressions through fuzzy level factors which improves results in extracting mixed feelings. BharatNatyam, a renowned Indian Classical Dance was taken as Dataset for test verification.

V. Gomathi, Dr. K. Ramar, and A. Santhiyaku Jeevakumar [10], used neuro-fuzzy based automatic facial expression recognition system to recognize the human facial expressions like happy, fear, sad, angry, disgust and surprise.

Initially facial image is segmented into three regions from which the uniform Local Binary Pattern (LBP) texture features distributions are extracted and represented as a histogram

(3)

International Journal of Advanced Engineering Science and Technological Research (IJAESTR) ISSN: 2321-1202, www.aestjournal.org @2016 All rights reserved

3

descriptor. The facial expressions are recognized using Multiple Adaptive Neuro Fuzzy Inference System (MANFIS).

The proposed system was designed and tested with JAFFE face database. The proposed model reports 94.29% of classification accuracy.

Shubhangi Giripunje and Preeti Bajaj [11], considered five different facial expressions. They applied firstly logarithmic Gabor filters for extracting the features. Then they selected the optimal subsets of features, for each expression.

The classification tasks were performed using the Neural Network. Secondly, this study indicates that the YALE database contains expressers that expressed expressions. This exploratory study aims at investigating the effects of terrorism to recognize emotions. They presented a view-based approach to the representation and recognition of human facial expression.

James A. Coan and John J.B. Allen [12] used an approach that analyses highly contaminated brain signals.

They extract relevant features for the emotion detection task based on neuroscience findings. They reached an average accuracy of 51%, 53%, 58% and 61% for joy, anger, fear and sadness, respectively. They approached on fewer number of electrodes that ranges from 4 to 25 electrodes and reached an average classification accuracy of 33% for joy emotion, 38%

for anger, 33% for fear and 37.5% for sadness using 4 or 6 electrodes only.

III. EXTRACTION OF FACIAL FEATURE

The face of a human has several features such as, mouth, eyes, nose, eyebrows, and forehead. Each of this features has a unique shape and a unique pattern, hence, many experiments have been reported in extracting facial feature for recognizing facial expression. Yh- yeong Chang et.al, used eyebrows, eyes, and mouth for facial expression labeling. We extract the features toward forehead wrinkle, mid forehead Wrinkle, cheek wrinkle, and mouth length as seen in Fig.2[14]

Fig.2. Parts of Facial Feature Extracted

Line Face Detection: Facial features extracted using edge detection and morphology technique to obtain the lines on the face. We used canny edge detection. Before applying Canny edge detection method, at first, the images should be optimized using brightness and contrast tuning as shown in Figure 3[14]. The aim of this optimizing is to detect the vague lines such as the face wrinkle.

Fig. 3.The image processing (a) Canny edge detection with no contrast and brightness tuning, (b)Canny edge detection with contrast and brightness tuning

IV. TRAINING AND BACK PROPAGATION NEURAL NETWORK

We use back-propagation algorithm to recognize of facial expression with feed-forward architecture. Back-Propagation neural networks are the most widely used network and are considered the work horse of artificial neural network. It can be used to model complex relationships between inputs and outputs or to find patterns in data. The back-propagation of feed-forward architecture is designed based on facial features extracted as illustrated in Figure 4. It consists of an input layer containing four neurons representing input variable to the problem, that is extracted data from the forehead wrinkle, the mid forehead wrinkle, the cheek wrinkle, and the mouth length; one hidden layers containing one or more neurons to help capture the nonlinearity in the data; and an out-put layer containing six nodes representing output variable to the problem, that is facial expressions: anger, disgust, surprise, happiness, sadness and fear. The neurons between layers are fully interconnected with weight vij and wij.

(4)

International Journal of Advanced Engineering Science and Technological Research (IJAESTR) ISSN: 2321-1202, www.aestjournal.org @2016 All rights reserved

4

Fig. 4 Architecture of feed-forward back-propagation

neural network for facial expression recognition The training of a network by back-propagation neural network involves three stages: the feed-forward of the input training pattern, the calculation and back-propagation of associated error, and the adjustment of the weights. The data are fed forward from the input layer, through hidden layer, to output layer without feedback. Then, based on the feed-forward error back-propagation learning algorithm, back-propagation will search the error surface using gradient descent for point(s).

Based on the error, the portion of error correction is computed, and then the weights for all layers are adjusted simultaneously.

In many neural network applications, the data (input or target patterns) have the same range of values. We use the binary sigmoid function, which has range of (0, 1) and is defined as f(x) = 1/ (1 + exp (-x)), that's why the data is also represented in binary form or has range of 0-1. The representation data of input is explained in the section before. Table.I shows the data of training pairs (input and target patterns) in back propagation of neural network. We use two pairs of training input data for each of six output expressions. The first row is for neutral expression.

TABLE I.THE DATA OF TRAINING PAIRS IN BACKPROPAGATION IN NN

V. RECOGNITION

Once the training is over, the network is ready to recognize gesture presented at its input. For recognizing the expression of a face two options are provided. If user wants to recognize the gesture of existing image, then it can be loaded from memory. As the user selects the image, the face recognition method works and returns the face part of the image. And other option is to capture the live image. Image is captured from the web cam. For testing purpose “Logitech Quick Cam Pro” is used. Once asked to recognize, it captures the image and finds the face part in it. Then the edge detection, thinning, and token generation are performed. Then it classifies the given tokens into one of three gestures it learned during training. It gives percentage of recognition to each gesture with highest percentage closely matching and lowest to the farthest matching and the closest match is considered as the result. The recognition process is implemented as per the outline given in the flow chart in Figure 5.

(5)

5

Fig.5 Recognition Process

VI. CONCLUSION

The success of the abstract message of facial expression being recognizable by the computer demonstrates that other kinds of abstract information can still be recognized by a computer, which will result in much broader applications of the computer. Using neural network model of back propagation algorithm to train neural networks is a viable approach. Novel methods of performance tuning of learning algorithm are possible when using this model.

There is, however, a risk of learning instability with this approach, and actual modeling of back propagation can be done in several different modes.

REFERENCES

[1] P. Ekman, W.V. Friesen, “Constants across cultures in the face and emotion”, Journal of Personality and Social Psychology, Vol 17(2), 124–129.1 971.

[2] S. Ravi and Mahima S., “Study of the Changing Trends in Facial Expression Recognition”, International Journal of Computer Applications (0975 – 8887), Volume 21– No.5, May 2011.

[3] Abu Sayeed Md. Sohail, and Prabir Bhattacharya,

“Classifying Facial Expressions Using Point-Based Analytic Face Model and Support Vector Machines”, IEEE transactions on Systems, Man & Cybernetics, 1008-1013, 2007.

[4] Jen-Chun Lin, Chung-Hsien Wu, and Wen-Li Wei, “Error Weighted Semi-Coupled Hidden Markov Model for Audio- Visual Emotion Recognition”, IEEE Transactions on Multimedia, vol. 14, no. 1, February 2012.

[5] Ratliff Matthew S. and Patterson Eric, “Emotion Recognition Using Facial Expressions with Active Appearance Models", Third International Conference on Human Computer Interaction. ACTA Press Anaheim, CA, USA (2008).

[6] Raheja J., Kumar U., “Human Facial Expression Detection from Detected in Captured Image Using Back propagation Neural Network”, International Journal of Computer Science & Information technology (IJCSIT), Vol.2. No.1, (2010).

[7] Karthigayan M., Rizon M., Nagarajan R. and Yaacob S.,

“Genetic Algorithm and Neural

Network for Face Emotion Recognition”, University of Malaysia Perlis (UNIMAP), (2008).

[8] N. M. Hewahi, A.R. M. Baraka , “Impact of Ethnic Group on Human Emotion Recognition Using Back propagation Neural Network”, BRAIN. Broad Research in Artificial Intelligence and Neuroscience Volume 2, Issue 4, December 2011, ISSN 2067-3957 (online), ISSN 2068 - 0473 (print).

[9] A. Butalia, Dr. A.K. Ramani, Dr. Maya Ingle, Dr. Parag Kulkarni, “Classification of Sentiments through Rough Fuzzy Approach”, IRACST - International Journal Of Computer Science And Information Technology & Security (IJCSITS), ISSN: 2249-9555,Vol. 2, No.2, April 2012.

[10] V. Gomathi, Dr. K. Ramar, and A. Santhiyaku Jeevakumar, “Human Facial Expression Recognition using MANFIS Model”, World Academy of Science, Engineering and Technology 50 .2009.

[11] Shubhangi Giripunje and Preeti Bajaj, “Recognition of Facial Expressions for Images using Neural Network”, International Journal of Computer Applications (0975 – 8887) Volume 40– No.11, February 2012.

[12] James A. Coan and John J.B. Allen , “Using minimal number of electrodes for emotion detection using brain signals produced from a new elicitation technique”, Int. J.

Autonomous and Adaptive Communications Systems, Vol. 6, No. 1, 2013.

(6)

6

[13] Elfenbein, H. A., & Ambady, N. (2002a). On the universality and cultural specificity of emotion recognition: A meta-analysis. Psychological Bulletin, 128(2), 205-235.

[14] Neeraj Shukla and Anuj Kumar, “Using Back- Propagation Recognition of Facial Expression” Journal of Environmental Science, Computer Science and

Engineering & Technology, Vol.2.No.1, 39-45. December 2012 -February 2013