Emotion

After analysing existing techniques for stress detection in Mobile SSP, this subsection focuses on emotion detection. To detect emotion in a preliminary stage, researchers

perform some simplification by focussing on the identification of major emotions such as happiness, anger, neutral, sadness etc. or just classifying if the user has positive or negative emotions. For emotion detection, scientists utilised audio datasets targeting different emotions and used them as training sets for machine learning techniques. Next, state-of-the-art techniques researchers utilised will be presented and a brief discussion about them will be provided.

At first, AMMON [57] extracted prosodic and spectral features from Belfast Naturalis- tic Database [158], and trained an SVM [157] classifier with 75% accuracy for emotion recognition i.e. positive or negative. An important work is EmotionSense [41] which used Speech and Transcripts library [159] to train an emotion recognition model and succeeded in 71% accuracy for 5 emotions based on prosodic features. Visage [113] detected users’ emotion on mobile phones through facial expression detection [128]. To evaluate their approach, they applied it on the JAFFE dataset [160] achieving the corresponding accuracies: a) anger 82.16%, b) disgust 79.68%, c) fear 83.57%, d) hap- piness 90.30%, e) neutral 89.93%, f) sadness 73.24% and g) surprise 87.52%. In [118] they apply facial expression classification to detect a user’s emotion and discriminate among four different emotions: a) neutral, b) joy, c) sad, d) surprise.

Discussion. As mentioned above, AMMON focused only on extracting information regarding the users having a positive or negative emotion, which induces some general- isation. By performing classification with several feature sets, they achieved acceptable accuracy given the trade-off of computational load when including glottal timings in the feature set. Formant tracking including Newton-Raphson method is a high work- load process, while in case the eigensolver fails additional burden is created by the construction of Toepliz matrices. FFT is another technique that is computationally expensive and should be considered before being applied on a mobile device intended for continuous inference.

EmotionSense includes components for adapting the sensing process based on the con- text. It showcases the effects in computation, communication cost and energy for performing the computations on the device or on a backend server. Authors trained the emotion detection model on a state-of-the-art library. However, there is a need

to evaluate this model not only based on the trained library but also in a real-world environment to understand the robustness of the model. They performed speaker recog- nition on samples retrieved from 10 users. But there is no indication in what type of environment the data were collected from i.e. indoor, outdoor, with(out) ambient noise etc. Furthermore, adding Brownian noise is not sufficient to prove that the detection model is able to tolerate noise introduced by real-world environments. Similarly, the emotion recognition model was only evaluated on data from the training library. In essence, providing an evaluation of each of the components (speaker, emotion recog- nition) individually and as a holistic approach on real-world data, would indicate the robustness of the system in daily life monitoring. This necessitates the conduction of a larger-scale experiment for further analysis.

Visage utilises a well-established, robust and accurate method for face recognition com- bined with the device’s orientation. However, this approach requires the user to hold the device in a position so as the mobile phone’s camera is targeting the user’s face. The face recognition approach through Fisherfaces [128] provides tolerance in variations of lightning and expressions in comparison to other techniques such as Eigenfaces [117]. Also, it should be noted that the system operates in a supervised manner. Thus, it re- quires from the user to provide predefined facial expressions to construct a personalised model that classifies the seven distinct emotions.

In [118] authors were able to achieve a reasonable emotion recognition accuracy (70- 80%) for four emotions. They utilised a boosted Na¨ıve Bayes for classification which introduces a certain computation load in the training process due to the creation of domain specific classifiers. Likewise, this approach is prone to the creation of domain specific classifiers for possible outliers, inducing over-fitting. The system requires pre- loaded images in the device and does not support real-time recognition of user’s emotion through facial expressions.

Based on the above techniques certain parameters should be considered. The highest accuracy was achieved through facial expression recognition in Visage. However, it induces intrusiveness due to the requirement that the device’s camera should target user’s face. Also, the computational burden induced by face recognition and facial

point tracking must be considered. EmotionSense managed an acceptable accuracy in an energy efficient manner, without requiring a specific on-body position of the device or any external hardware. AMMON provided only a preliminary classification result regarding the user’s emotion but based on the application could be utilised. Regarding [118], the restricted inference context of the application indicates it as a less qualified system with respect to the others, for continuous sensing and inference.