Feature smoothing

In the last experiment, the smoothing methods were compared among each other. The smooth- ing method was fixed and the landmark algorithms and registration method were combined.

The aim was to reduce the noise of the descriptors that might be caused by e.g. inaccurate facial rectification.

On all three plots Fig.7.12,7.13and7.14can be seen that the feature smoothing has positive impact on the accuracy. Moving average and polynomial function fitting have almost the same influence on the AUC measure on all three databases.

7.4 Introspection experiments withoutRegistration-intrafacewithoutRegistration-chehra rotationAndTranslation-intrafacerotationAndTranslation-chehra similarity-intraface similarity-chehra phaseCorrelation-intrafacephaseCorrelation-chehra 0.75 0.8 0.85 0.9 AUC none moving_avg polynomial

Figure 7.13 Comparison of the area under the ROC curve (AUC) among different feature smoothing

methods for combinations of registration methods and landmark algorithms in the pipeline for the database SMIC E HS. withoutRegistration-intraface rotationAndTranslation-intraface similarity-intraface phaseCorrelation-intraface 0.76 0.78 0.8 0.82 0.84 0.86 AUC none moving_avg polynomial

Figure 7.14 Comparison of the area under the ROC curve (AUC) among different feature smoothing

7 Experiments

Table 7.1 Results of the pipeline for the CAME2 database of the baseline method.

# Method AUC 1 CASME2-moving avg-rotationandtranslation-chehra 0.96263 2 CASME2-parabolic-rotationandtranslation-chehra 0.96144 3 CASME2-parabolic-similarity-chehra 0.95649 4 CASME2-moving avg-similarity-chehra 0.95592 5 CASME2-parabolic-rotationandtranslation-intraface 0.95371 6 CASME2-moving avg-rotationandtranslation-intraface 0.95049 7 CASME2-parabolic-fft-chehra 0.9504 8 CASME2-moving avg-fft-chehra 0.94777 9 CASME2-parabolic-similarity-intraface 0.9459 10 CASME2-parabolic-withoutRegistration-chehra 0.94356 11 CASME2-parabolic-withoutRegistration-intraface 0.94347 12 CASME2-moving avg-withoutRegistration-chehra 0.94335 13 CASME2-moving avg-withoutRegistration-intraface 0.94279 14 CASME2-moving avg-similarity-intraface 0.94138 15 CASME2-parabolic-fft-intraface 0.93015 16 CASME2-none-rotationandtranslation-chehra 0.9249 17 CASME2-moving avg-fft-intraface 0.92385 18 CASME2-none-similarity-chehra 0.91576 19 CASME2-none-rotationandtranslation-intraface 0.90603 20 CASME2-none-withoutRegistration-intraface 0.90359 21 CASME2-none-withoutRegistration-chehra 0.90189 22 CASME2-none-fft-chehra 0.896 23 CASME2-none-similarity-intraface 0.89047 24 CASME2-none-fft-intraface 0.87381

7.4 Introspection experiments

Table 7.2 Results of the pipeline for the SMIC HS database of the baseline method.

# Method AUC

1 SMIC HS-parabolic-rotationandtranslation-chehra 0.90181 2 SMIC HS-moving avg-rotationandtranslation-chehra 0.89504 3 SMIC HS-parabolic-similarity-chehra 0.89279 4 SMIC HS-moving avg-similarity-chehra 0.88923 5 SMIC HS-parabolic-rotationandtranslation-intraface 0.88564 6 SMIC HS-moving avg-rotationandtranslation-intraface 0.88374 7 SMIC HS-none-rotationandtranslation-chehra 0.88001 8 SMIC HS-parabolic-similarity-intraface 0.86822 9 SMIC HS-moving avg-similarity-intraface 0.86601 10 SMIC HS-none-similarity-chehra 0.86089 11 SMIC HS-none-rotationandtranslation-intraface 0.85291 12 SMIC HS-parabolic-withoutRegistration-chehra 0.84594 13 SMIC HS-moving avg-withoutRegistration-chehra 0.83787 14 SMIC HS-none-similarity-intraface 0.83574 15 SMIC HS-parabolic-withoutRegistration-intraface 0.8323 16 SMIC HS-moving avg-withoutRegistration-intraface 0.83021 17 SMIC HS-parabolic-fft-chehra 0.81203 18 SMIC HS-moving avg-fft-chehra 0.80637 19 SMIC HS-parabolic-fft-intraface 0.797 20 SMIC HS-moving avg-fft-intraface 0.79397 21 SMIC HS-none-withoutRegistration-chehra 0.7924 22 SMIC HS-none-withoutRegistration-intraface 0.77993

23 SMIC HS-none-fft-chehra 0.75432

24 SMIC HS-none-fft-intraface 0.74447

Table 7.3 Results of the pipeline for the MEVIEW database of the baseline method.

# Method AUC 1 MEVIEW-parabolic-similarity 0.85604 2 MEVIEW-parabolic-withoutRegistration 0.85324 3 MEVIEW-moving avg-withoutRegistration 0.85119 4 MEVIEW-moving avg-similarity 0.84619 5 MEVIEW-none-withoutRegistration 0.83477 6 MEVIEW-parabolic-rotationandtranslation 0.82759 7 MEVIEW-moving avg-rotationandtranslation 0.82621 8 MEVIEW-parabolic-fft 0.82242 9 MEVIEW-moving avg-fft 0.82185 10 MEVIEW-none-similarity 0.81858 11 MEVIEW-none-rotationandtranslation 0.79667 12 MEVIEW-none-fft 0.76882

8 Conclusion

In the thesis, we studied a specific type of facial expressions called micro-expressions. The importance of the micro-expression detection is that micro-expressions are involuntary thus reveal the true emotions of a person. Therefore, having a good detection method might be useful for criminal investigation, airport security or psychological examination.

A simple detection method was proposed. The method was designed to spot MEs in non- laboratory conditions, but also in real-world videos. The method is based on measuring the brief local image intensity changes caused by facial muscle contractions, which result in a specific pattern.

Despite few micro-expressions databases were already published, all of them were recorded in laboratory conditions with a similar protocol and are mostly designed for the isolated recog- nition. It was desired to find another way of obtaining micro-expressions, preferably in the real situations.

We collected the MEVIEW dataset of “in-the-Wild” phenomena, consisting of mostly poker tournament videos downloaded from YouTube. Poker games seem to be rich in micro-expression occurrences as the basic assumptions, e.g. stress, fear of loss, are fulfilled.

The proposed method was evaluated on SMIC and CASMEII databases and our challenging MEVIEW database. The proposed algorithms were compared against a method of [27] with favorable results. An introspection throughout the designed choices was carried out.

As a limitation of the proposed method, we see the number of false alarms. Nevertheless, we observed the correct micro-expressions tend to give very high scores. More examples collected by manual inspection of high scoring events in long videos by experienced human annotators could be important and more data would surely allow designing a more sophisticated classifier with higher detection accuracy.

Bibliography

[1] Muscles of facial expression. https://www.kenhub.com/en/library? sequence=facial-muscles. Accessed: 2016-12-19. 5,6

[2] Akshay Asthana, Stefanos Zafeiriou, Shiyang Cheng, and Maja Pantic. Incremental face alignment in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1859–1866, 2014. 15,36

[3] Radom´ır ˇCIH ´AK. Anatomie 2. 2. vyd. Praha. Grada Publishing, 2002. 6

[4] Victoria Contreras. Artnatomya.http://www.artnatomia.net/, 2005. 6

[5] Charles Darwin, Paul Ekman, and Phillip Prodger. The expression of the emotions in man and animals. Oxford University Press, USA, 1998. 5

[6] A. K. Davison, C. Lansley, N. Costen, K. Tan, and M. H. Yap. Samm: A spontaneous micro-facial movement dataset. IEEE Transactions on Affective Computing, PP(99):1–1, 2016. 12,26,29,30

[7] Adrian K Davison, Moi Hoon Yap, and Cliff Lansley. Micro-facial movement detection using individualised baselines and histogram-based descriptors. In Systems, Man, and Cybernetics (SMC), 2015 IEEE International Conference on, pages 1864–1869. IEEE, 2015. 12

[8] Paul Ekman. Darwin, deception, and facial expression. Annals of the New York Academy of Sciences, 1000(1):205–221, 2003. 10

[9] Paul Ekman. Micro Expressions Training Tool. Emotionsrevealed. com, 2003.10,12,30

[10] Paul Ekman. Emotions revealed: Recognizing faces and feelings to improve communica- tion and emotional life. Macmillan, 2007.11

[11] Paul Ekman. Lie catching and microexpressions. The philosophy of deception, pages 118–133, 2009. 10

[12] Paul Ekman and Wallace V Friesen. Nonverbal leakage and clues to deception. Psychia- try, 32(1):88–106, 1969. 10,26

[13] Paul Ekman and Wallace V Friesen. Facial action coding system. 1977. 6,7,11,16

[14] Paul Ekman and Wallace V Friesen. Unmasking the face: A guide to recognizing emotions from facial clues. Ishk, 2003. 7,10

Bibliography

[15] Paul Ekman, Wallace V Friesen, and Phoebe Ellsworth. Emotion in the human face: Guidelines for research and an integration of findings. Elsevier, 2013.5

[16] Paul Ekman and Dacher Keltner. Universal facial expressions of emotion. California Mental Health Research Digest, 8(4):151–158, 1970. 5

[17] Paul Ekman, Maureen O’Sullivan, and Mark G Frank. A few can catch a liar. Psycholog- ical science, 10(3):263–266, 1999. 10

[18] Margaret J Fehrenbach and Susan W Herring. Illustrated anatomy of the head and neck. Elsevier Health Sciences, 2015. 5,6

[19] Mark G Frank and Paul Ekman. The ability to detect deceit generalizes across different types of high-stake lies. Journal of personality and social psychology, 72(6):1429, 1997.

10

[20] Mark G Frank, Carl J Maccario, and Venugopal Govindaraju. Behavior and security. Protecting Airline Passengers in the Age of Terrorism. Greenwood Pub Group, Santa Barbara, California, pages 86–106, 2009. 10

[21] Wallace V Friesen and Paul Ekman. Emfacs-7: Emotional facial action coding system. Unpublished manuscript, University of California at San Francisco, 2(36):1, 1983. 7

[22] Colin Goodall. Procrustes methods in the statistical analysis of shape. Journal of the Royal Statistical Society. Series B (Methodological), 53(2):285–339, 1991. 16

[23] A Ardeshir Goshtasby. Image registration: Principles, tools and methods. Springer Science & Business Media, 2012. 18

[24] Yanjun Guo, Yantao Tian, Xu Gao, and Xuange Zhang. Micro-expression recognition based on local binary patterns from three orthogonal planes and nearest neighbor method. In 2014 International Joint Conference on Neural Networks (IJCNN), pages 3473–3479. IEEE, 2014. 13

[25] Ernest A Haggard and Kenneth S Isaacs. Micromomentary facial expressions as indicators of ego mechanisms in psychotherapy. In Methods of research in psychotherapy, pages 154–165. Springer, 1966. 10

[26] Carl-Herman Hjortsj¨o. Man’s face and mimic language. Studen litteratur, 1969. 5

[27] Xiaobai Li, Xiaopeng Hong, Antti Moilanen, Xiaohua Huang, Tomas Pfister, Guoying Zhao, and Matti Pietik¨ainen. Reading hidden emotions: spontaneous micro-expression spotting and recognition. arXiv preprint arXiv:1511.00423, 2015. 12,32,33,34,35,44

[28] Xiaobai Li, T. Pfister, Xiaohua Huang, Guoying Zhao, and M. Pietikainen. A spontaneous micro-expression database: Inducement, collection and baseline. In Automatic Face and Gesture Recognition (FG), 2013 10th IEEE International Conference and Workshops on, pages 1–6, April 2013.12,26,27,32

Bibliography

[29] David Matsumoto and Hyi Sung Hwang. Evidence for training the ability to read mi- croexpressions of emotion. Motivation and Emotion, 35(2):181–191, 2011. 10

[30] David Matsumoto, Seung Hee Yoo, and Sanae Nakagawa. Culture, emotion regulation, and adjustment. Journal of personality and social psychology, 94(6):925, 2008. 10

[31] Antti Moilanen, Guoying Zhao, and Matti Pietik¨ainen. Spotting rapid facial movements from videos using appearance-based feature difference analysis. In Pattern Recognition (ICPR), 2014 22nd International Conference on, pages 1722–1727. IEEE, 2014.12

[32] Maureen O’Sullivan, Mark G Frank, Carolyn M Hurley, and Jaspreet Tiwana. Police lie detection accuracy: The effect of lie scenario. Law and Human Behavior, 33(6):530–538, 2009. 10

[33] Devangini Patel, Guoying Zhao, and Matti Pietik¨ainen. Spatiotemporal integration of optical flow vectors for micro-expression detection. In International Conference on Ad- vanced Concepts for Intelligent Vision Systems, pages 369–380. Springer, 2015. 12

[34] Tomas Pfister, Xiaobai Li, Guoying Zhao, and Matti Pietik¨ainen. Recognising sponta- neous facial micro-expressions. In 2011 International Conference on Computer Vision, pages 1449–1456. IEEE, 2011.12,13

[35] S. Polikovsky, Y. Kameda, and Y. Ohta. Facial micro-expressions recognition using high speed camera and 3d-gradient descriptor. In Crime Detection and Prevention (ICDP 2009), 3rd International Conference on, pages 1–6, Dec 2009. 12,13,16,17,29

[36] Senya Polikovsky and Yoshinari Kameda. Facial micro-expression detection in hi-speed video based on facial action coding system (facs). IEICE transactions on information and systems, 96(1):81–92, 2013. 12,13

[37] Xun-bing Shen, Qi Wu, and Xiao-lan Fu. Effects of the duration of expressions on the recognition of microexpressions. Journal of Zhejiang University Science B, 13(3):221– 230, 2012.10

[38] Matthew Shreve, Sridhar Godavarthy, Dmitry Goldgof, and Sudeep Sarkar. Macro-and micro-expression spotting in long videos using spatio-temporal strain. In Automatic Face & Gesture Recognition and Workshops (FG 2011), 2011 IEEE International Conference on, pages 51–56. IEEE, 2011. 12,30

[39] Su-Jing Wang, Wen-Jing Yan, Xiaobai Li, Guoying Zhao, Chun-Guang Zhou, Xiaolan Fu, Minghao Yang, and Jianhua Tao. Micro-expression recognition using color spaces. IEEE Transactions on Image Processing, 24(12):6034–6047, 2015. 13

[40] Sujing Wang, Wen-Jing Yan, Xiaobai Li, Guoying Zhao, and Xiaolan Fu. Micro- expression recognition using dynamic textures on tensor independent color space. In ICPR, pages 4678–4683. Citeseer, 2014. 13

Bibliography

[41] Sharon Weinberger. Airport security: Intent to deceive? Nature, 465(7297):412–415, 2010. 10

[42] Qi Wu, Xunbing Shen, and Xiaolan Fu. The machine knows what you are hiding: an automatic micro-expression recognition system. In International Conference on Affective Computing and Intelligent Interaction, pages 152–162. Springer, 2011. 12,30

[43] Xuehan Xiong and Fernando De la Torre. Supervised descent method and its applications to face alignment. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2013. 15,36

[44] Feng Xu, Junping Zhang, and James Wang. Microexpression identification and catego- rization using a facial dynamics map. 2016. 13

[45] Wen-Jing Yan, Xiaobai Li, Su-Jing Wang, Guoying Zhao, Yong-Jin Liu, Yu-Hsin Chen, and Xiaolan Fu. Casme ii: An improved spontaneous micro-expression database and the baseline evaluation. PloS one, 9(1):e86041, 2014. 12,28,29,32

[46] Wen-Jing Yan, Qi Wu, Jing Liang, Yu-Hsin Chen, and Xiaolan Fu. How fast are the leaked facial expressions: The duration of micro-expressions. Journal of Nonverbal Behavior, 37(4):217–230, 2013. 10

[47] Wen-Jing Yan, Qi Wu, Yong-Jin Liu, Su-Jing Wang, and Xiaolan Fu. Casme database: A dataset of spontaneous micro-expressions collected from neutralized faces. In Auto- matic Face and Gesture Recognition (FG), 2013 10th IEEE International Conference and Workshops on, pages 1–7, April 2013. 12,26,28