Emotion estimation of people wearing masks using machine learning
DOI:
https://doi.org/10.15837/ijccc.2024.1.5363Keywords:
KNN, CNN, deep learning, emotion estimation, masked face, SVMAbstract
Coronavirus (COVID-19) is the infectious agent responsible for the transmission of SARS (severe acute respiratory syndrome). When an infected person coughs, talks, or breathes, it spreads as small droplets of fluid from the mouth and nose of the infected person. Sanitary masks help prevent the spread of the virus from the person wearing the mask to others. This new behavior may cause a number of problems in interpersonal interactions. The goal of this paper is the emotion estimation of masked faces. It presents two major parts. At the first level, we created a system for identifying sanitary masks, using CNN. At the second one, we developed an emotion estimation system in order to estimate the classification rates of a masked face. It consists of three steps: face element detection, feature point localization, and classification. We used the well-known Viola and Jones algorithm in order to achieve the first step. We used several techniques to estimate emotions (SVM, KNN and deep learning). We made comparisons of the obtained results. Particular attention is also given to the effect of face masks on the performance of various methods.References
Rim Afdhal, Ridha Ejbali , Mourad Zaied and Chokri Ben Amar. Emotion recognition using features distances classified by wavelets network and trained by fast wavelets transform. International Conference on Hybrid Intelligent Systems(HIS'14) proceedings, Golf University for Science and Technologie, Kuwait. pp.238,241. December 14-16., 14h.
C. Darwin. The expression of the emotions in man and animals.1872.
https://doi.org/10.1037/10001-000
P. Ekman. Are there basic emotions? Psychological Review, 99 :550-553,1992.
https://doi.org/10.1037/0033-295X.99.3.550
C.E Izard. Human emotions. Plenum Press New York, 1977.
https://doi.org/10.1007/978-1-4899-2209-0
R. Afdhal, R. Ejbali and M. Zaied. Emotion recognition using the shapes of the wrinkles. International Conference on Computer and Information Technology, December 18-20, 2016.
https://doi.org/10.1109/ICCITECHN.2016.7860193
Rim Afdhal, Ridha Ejbali and Mourad Zaied. A Hybrid System Based on Wrinkles Shapes and Biometric Distances for Emotion Recognition. International Conference on Advances in Computer-Human Interactions,ACHI 2017 .
Ahmed A. Elngar, Mohamed Arafa, Abd El Rahman Ahmed Naeem, Ahmed Rushdy Essa, Zahra Ahmed shaaban. The Viola-Jones Face Detection Algorithm Analysis: A Survey. Journal of Cybersecurity and Information Management (JCIM),April 2021.
Tanmoy Paul, Ummul Afia Shammi, Mosabber Uddin ahmed, Rashedur Rahman, Syoji Kobashi and Md Atiqur Rahman Ahad. A Study on Face Detection Using Viola-Jones Algorithm for Various Backgrounds, Angels and Distances. Human Activity Recognition,May 2018.
Wang Kay Ngai, Haoran Xieb, Di Zouc, Kee Lee Chou. Emotion Recognition based on Convolutional Neural Networks and Heterogeneous Bio-signal Data Source. Journal Pre-proof, 25 July 2021.
Noyes, E.; Davis, J.P.; Petrov, N.; Gray, K.L.H.; Ritchie, K.L.The effect of face masks and sunglasses on identity and expression recognition with super-recognizers and typical observers.R. Soc. Open Sci. 2021, 8, 201169. [Google Scholar] [CrossRef].
https://doi.org/10.1098/rsos.201169
Bassili, J. N.Emotion recognition: The role of facial movement and the relative importance of upper and lower areas of the face. Journal of Personality and Social Psychology,37(11), 2049-2058, (1979).
https://doi.org/10.1037/0022-3514.37.11.2049
Melina Grahlow , Claudia Rupp3, Birgit Derntl. The impact of face masks on emotion recognition performance and perception of threat. Journal of Personality and Social ,10.1371/journal. pone.0262840, 2022; 17(2): e0262840.
https://doi.org/10.1371/journal.pone.0262840
Claus-Christian Carbon. Wearing Face Masks Strongly Confuses Counterparts in Reading Emotions. September 2020.
Marco Marini1, AlessandroAnsani, Fabio Paglieri1 , Fausto Caruana & MarcoViola. The impact of facemasks on emotion recognition, trust attribution and re-identifcation. 10 March 2021.
https://doi.org/10.1038/s41598-021-84806-5
Pamela Parada-Fernandez, David Herrero-Fernandez, Richard Jorge, Patricia Comesana.Wearing mask hinders emotion recognition, but enhances perception of attractiveness . Personality and Individual Differences Volume 184, January 2022, 111195.
https://doi.org/10.1016/j.paid.2021.111195
Carbon, C.-C. Wearing Face Masks Strongly Confuses Counterparts in Reading Emotions. . Psychol. 2020, 11, 566886. [Google Scholar] [CrossRef].
https://doi.org/10.3389/fpsyg.2020.566886
Blais, C.; Roy, C.; Fiset, D.; Arguin, M.; Gosselin, F. The eyes are not the window to basic emotions. Neuropsychologia 2012, 50, 2830-2838. [Google Scholar] [CrossRef].
https://doi.org/10.1016/j.neuropsychologia.2012.08.010
Henrik Nordström. Emotional Communication in the Human Voice. Doctoral Thesis in Psychology at Stockholm University, Sweden 2019 , Sweden 2019.
Ram Kumar Madupu; Chiranjeevi Kothapalli; Vasanthi Yarra; Sonti Harika; Cmak Zeelan Basha. Automatic Human Emotion Recognition System using Facial Expressions with Convolution Neural Network. International conference of Electronics, Communication and Aerospace Technology (ICECA) , 05-07 November 2020.
https://doi.org/10.1109/ICECA49313.2020.9297483
Melina Grahlow , Claudia Rupp3, Birgit Derntl. The impact of face masks on emotion recognition performance and perception of threat. Journal of Personality and Social ,10.1371/journal. pone.0262840, 2022; 17(2): e0262840.
https://doi.org/10.1371/journal.pone.0262840
Ekman, P., Friesen, W. V., & Hager. Facial Action Coding System: facial action coding system: the manual: on CD-ROM. Salt Lake City, UT, USA: Research Nexus J. C. (2002).
Ahmad Alzu'bi, Firas Albalas, Tawfik AL-Hadhrami, Lojin Bani Younis and Amjad Bashayreh. Masked Face Recognition Using Deep Learning: A Review IoT Based Intelligent and Green Communications: Modelling, Practice and Applications. 31 October 2021.
https://doi.org/10.3390/electronics10212666
Mukhriddin Mukhiddinov, Oybek Djuraev, Farkhod Akhmedov, Abdinabi Mukhamadiyev and Jinsoo Cho. Masked Face Emotion Recognition Based on Facial Landmarks and Deep Learning Approaches for Visually Impaired People Application of Semantic Technologies in Sensors and Sensing Systems. 17 January 2023.
https://doi.org/10.3390/s23031080
Wegrzyn, M.; Vogt, M.; Kireclioglu, B.; Schneider, J.; Kissler, J. Mapping the emotional face. How individual face parts contribute to successful emotion recognition. PLoS ONE 2017, 12, e0177239. [Google Scholar] [CrossRef].
https://doi.org/10.1371/journal.pone.0177239
Beaudry, O.; Roy-Charland, A.; Perron, M.; Cormier, I.; Tapp, R. Featural processing in recognition of emotional facial expressions. Cogn. Emot. 2013, 28, 416-432. [Google Scholar] [CrossRef] [PubMed].
https://doi.org/10.1080/02699931.2013.833500
Schurgin, M.W.; Nelson, J.; Iida, S.; Ohira, H.; Chiao, J.Y.; Franconeri, S.L. Eye movements during emotion recognition in faces. J. Vis. 2014, 14, 14. [Google Scholar] [CrossRef] [PubMed].
https://doi.org/10.1167/14.13.14
Gosselin, F.; Schyns, P.G. Bubbles: A technique to reveal the use of information in recognition tasks. Vis. Res. 2001, 41, 2261-2271. [Google Scholar] [CrossRef].
https://doi.org/10.1016/S0042-6989(01)00097-9
Chih-Che Chung, Wei-Ting Lin, Rong Zhang, Kai-Wen Liang, and Pao-Chi Chang. Emotion Estimation by Joint Facial Expression and Speech Tonality Using Evolutionary Deep Learning Structures. 2019 IEEE 8th Global Conference on Consumer Electronics (GCCE) .
https://doi.org/10.1109/GCCE46687.2019.9015558
Andrea Francesco Abate, Lucia Cimmino, Bogdan-Costel Mocanu, Fabio Narducci, Florin Pop. The limitations for expression recognition in computer vision introduced by facial masks. Multimedia Tools and Applications 14 July 2022.
Additional Files
Published
Issue
Section
License
Copyright (c) 2023 Monia Hamdi, Naziha Hmidi, Ridha Ejbali, Mourad Zaied, Rim Afdhal
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
ONLINE OPEN ACCES: Acces to full text of each article and each issue are allowed for free in respect of Attribution-NonCommercial 4.0 International (CC BY-NC 4.0.
You are free to:
-Share: copy and redistribute the material in any medium or format;
-Adapt: remix, transform, and build upon the material.
The licensor cannot revoke these freedoms as long as you follow the license terms.
DISCLAIMER: The author(s) of each article appearing in International Journal of Computers Communications & Control is/are solely responsible for the content thereof; the publication of an article shall not constitute or be deemed to constitute any representation by the Editors or Agora University Press that the data presented therein are original, correct or sufficient to support the conclusions reached or that the experiment design or methodology is adequate.
