DEVELOPMENT OF A REAL-TIME EMOTION RECOGNITION SYSTEM USING FACIAL EXPRESSIONS AND EEG BASED ON MACHINE LEARNING METHODS AND DEEP NEURAL NETWORKS
Основное содержимое статьи
Аннотация
Real-time emotion recognition has been an active area of research over the past few decades. The purpose of this work is to classify the emotional expressions of children with physical disabilities (deaf, mute and bedridden) and children with autism based on facial landmarks and electroencephalograph (EEG) signals using convolutional neural network (CNN) and long short-term memory ( LSTM ) classifiers development of a real-time emotion recognition algorithm using virtual markers through an optical flow algorithm, which works effectively with uneven lighting and subject head rotation (up to 25°), different backgrounds and different skin tones. Six facial emotions (happiness, sadness, anger, fear, disgust and surprise) are collected using ten virtual markers. Fifty-five undergraduate students (35 men and 25 women) with an average age of 22.9 years volunteered to participate in a facial emotion recognition experiment. Nineteen undergraduate students volunteered to collect EEG signals. Initially, Haar-like features are used to detect faces and eyes. Later, virtual markers are placed at specific locations on the subject's face based on a facial action coding system using a mathematical model approach, and the markers are tracked using the Lucas-Kande optical flow algorithm. The distance between the center of the subject's face and the position of each marker is used as a feature to classify the facial expression. This distance feature was statistically confirmed using one-way ANOVA with a significance level of p < 0.01. In addition, fourteen signals collected from EEG signal readout channels ( EPOC ) are used as features for emotional classification using EEG signals. Finally, the features are cross-validated using five-fold cross-validation and fed to LSTM and CNN classifiers. We achieved a maximum recognition rate of 99.81% using CNN to detect emotions from facial landmarks. However, the maximum recognition rate achieved using the LSTM classifier is 87.25% for emotion detection from EEG signals.
Информация о статье
Библиографические ссылки
Dada E.G., Bassey J.S., Chiroma H., Abdulhamid S.M., Adetunmbi A.O., Ajibuwa O.E. Machine learning for email spam filtering: Review, approaches and open research issues. Helion 2019;5(6):e01802. https://doi.org/10.1016/j.heliyon.2019. e01802.
Hassouneh, Aya & mutawa, a.m & Murugappan, Prof. (2020). Development of a Real-Time Emotion Recognition System Using Facial Expressions and EEG based on machine learning and deep neural network methods. Informatics in Medicine Unlocked. 20. 100372. 10.1016/j.imu.2020.100372.
Xie M. Development of artificial intelligence and its influence on the financial system. J Phys Conf 2019;1187:032084. https://doi.org/10.1088/1742-6596/1187/3/032084.
Hegazi O., Soliman O.S., Salam M.A. Machine learning model for stock market forecasting. Int J Comput Sci Telecommun 2014;4(12):16–23.
Beckmann JS, Liu D. Reconciling evidence-based medicine and precision medicine in the era of big data: challenges and opportunities. Genome Med 2016;8(1):134–9.
Weber G.M., Mandl K.D., Kohane I.S. Finding the missing link for big biomedical data. Jama 2014;311(24):2479–80.
Loconsol S, Chiaradia D, Bevilacqua V, Frisoli A. Real-time emotion recognition: An improved hybrid approach for evaluating classification performance. Theory Intelligent Computing 2014: 320–31.
Huang H, Kortelainen J, Zhao G, Li H, Moilanen A, Seppanen T, Pietikainen M. Multimodal emotion analysis from facial expression and electroencephalogram. Comput-Visage-Image-Understanding 2016;147:114–24. https://doi.org/10.1016/j. cviu.2015.09.015.
Raheel A, Majeed M, Anwar SM. Facial expression recognition based on electroencephalography. In: 2019 2nd International Conference on Computing, Mathematics and Engineering Technology (iCoMET), Sukkur, Pakistan; 2019. p. 1–5.
Vasilis S., Herrmann Jurgen. Where do machine learning and human-computer interaction meet? 1997.
Keltiner D, Ekrman PW: Lewis M, Haviland Jones JM, editors. Emotion facial expression, hand drawn emotion book. New York: Guilford Press; 2000. p. 236–49.
Ekman P. Darwin and facial expression: A review of a century of research. United States of America: Ishq Academic Publishing House; 2006. p. 1973.
Ekman P., Friesen V.V. Constants in different cultures in face and emotions. J Pers Soc Psychol 1971;17(2):124.
Ekman P. Darwin and facial expression: A review of a century of research. United States of America: Ishq Academic Publishing House; 2006. p. 1973.
Ekman P., Friesen V.V., Ancoli S. Facial signs of emotional experience. J Pers Soc Psychol 1980;39:1123–34.
Ekman P., Friesen V.V., Ancoli S. Facial signs of emotional experience. J Pers Soc Psychol 1980;39:1123–34.
Nguyen BT, Trinh MH, Phan TV, Nguyen HD. Efficient,real-time emotion detection using camera and facial,landmarks. In: 2017 7th International Conference on Information Science and Technology (ICIST); 2017. https://doi.org/10.1109/icist.2017.7926765.
Loconsol S, Miranda SR, Augusto G, Frisoli A, Orvalo V. A new real-time emotion recognition method for geometric facial feature extraction. Proceedings of the International Conference on Theory and Applications of Computer Vision (VISAPP) 2014: 378–85.
Palestra Giuseppe, Pettinicchio Adriana, Del Coco Marco, Carcagne Pierluigi, Leo Marco, Distante Cosimo. Improved facial expression recognition performance using 32 geometric features. In: Proceedings of the 18th international conference on image analysis and processing. ICIAP; 2015. p. 518–28.
Zhang J, Yin Z, Cheng P, Nicole S. Emotion recognition using multimodal data and machine learning methods: a tutorial and review. Information synthesis. 2020.
Wilson PI, Fernandez J. Facial feature detection using Haar classifiers. J. Computer. Maly Sat., number 21, v. 2006;4. With. 127–133, ISSN 1937–4771.
Zhao G, Pietikasinen M. Dynamic texture recognition using local binary volume patterns. In: Vidal R, Hayden A, Ma Y, editors. Dynamic vision. WDV 2006, WDV 2005. Lecture Notes in Computer Science, vol. 4358. Berlin, Heidelberg: Springer; 2007.
Das P.K., Behera H.S., Pradhan S.K., Tripathi H.K., Jena P.K. Modified real-time A* algorithm and its performance analysis to improve path planning for a mobile robot. In: Computational Intelligence in Data Mining, Springer India, vol. 2; 2015. p. 221–34.
Viola P, Jones M. Fast object detection using an extended cascade of simple functions. 2001 [20] Aid, A. and Al-Hamadi, A. (2015). Improved human head pose estimation using the Kinect camera. In International Conference on Image Processing (ICIP), pages 1752–1756. IEEE.
Sally Judith D., Paul Sally. "Chapter 3: Pythagorean Triples". Roots of research: vertical development of mathematical problems. American Mathematical Society Bookstore; 2007. p. 63. ISBN 0821844032.
Sun Y, Chen Y, Wang H, Tang X. Deep face representation learning via joint identification-verification. In: Proc. Adv. Neural inf. Process. Syst.; 2014. p. 1988–96.
Ouyang W, Wang X, Zeng X, Qiu X, Luo P, Tian Y, Li H, Yang X, Wang Z, Loy CC, Tang X, Dong X, Loy CC, He K. Deepid-net: deformable deep convolutional neural networks for object detection. Q: In proc. IEEE Conf. Calculate. Vis. Pattern recognition; 2015. p. 2403–12.
Tang X. Image super-resolution using deep convolutional networks. IEEE Trans Pattern Anal Mach Intell 2016;38(2):295–307. https://doi.org/10.1109/tpami.2015.2439281.
Krizhevsky A., Sutskever I., Hinton G. ImageNet classification with deep convolutional neural networks. in Proc. Adv. Neural inf. Process. Syst. 2012: 1097–105.
Lang PJ, Bradley MM, Cuthbert BN. International Affective Picture System (IAPS): Affective picture ratings and instruction manual. Gainesville, FL: University of Florida; 2008. Technical Report A-8.
Bhattacharya J, Lindsen JP. Music for a brighter world: Brightness judgment biases due to musical emotion. PloS One 2016;11(2):e0148959. https://doi.org/10.1371/journal.pone.0148959.
Sangaya AK, Arumugam M, Bian GB. An intelligent learning approach to improve ECG signal classification and arrhythmia analysis. Artif Intell Med 2019: 101788. https://doi.org/10.1016/j.artmed.2019.101788.
Michel P, El Kaliouby R. Real-time facial expression recognition in video using support vector machines. In: Proc. 5th int. Conf. About multimodal interfaces; 2003. p. 258–64. Vancouver.
Suk M, Prabhakaran B. Mobile real-time facial expression recognition system—a case study. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops; Columbus, Ohio, USA; June 24–27, 2014 p. 132–7.
Jeong M, Ko BC. Real-time driver facial expression recognition for safe driving. Sensors 2018;18:4270.
Magdin Martin, Prickler F. Real-time facial expression recognition using webcam and SDK effectiva. International Journal of Interactive Multimedia and Artificial Intelligence 2017;1. https://doi.org/10.9781/ijimai.2017.11.002. In press.
Bahreini K, van der Vegt W, Vestera V. A fuzzy logic approach to robust real-time facial emotion recognition. Multimed Tool App 2019: 1–24.
Lucey P. et al. The extended Kohn-Kanade (CK) dataset: a complete dataset for action and emotional expression units. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops. IEEE; 2010. p. 94–101.