ENHANCING MICRO-EXPRESSION RECOGNITION: A NOVEL APPROACH WITH HYBRID ATTENTION-3DNET

(Received: 5-Sep.-2024, Revised: 11-Nov.-2024 , Accepted: 12-Nov.-2024)

Authors Budhi Irawan, Rinaldi Munir, Nugraha Priya Utama, Ayu Purwarianti,

Keywords #Micro-expression recognition #3D convolutional dual-path network #Hybrid attention #Squeeze-and-excitation blocks #Deep learning

Abstract This paper proposes a unique pipeline for micro-expression recognition using a Dual-path 3D Convolutional Neural Network enhanced with Hybrid Attention and Squeeze-and-Excitation Blocks. The three main goals of the pipeline are to (1) Optimize the extraction of spatial-temporal features using advanced neural network architectures, (2) Enhance data representation by implementing targeted image augmentation and balanced class distribution and (3) Enhance feature fusion using state-of-the-art network techniques. Comprehensive experiments were conducted on four benchmark datasets: CAS(ME)2, SMIC, SAMM and CASME II. The Hybrid Attention-3DNet model demonstrated superior recognition accuracy of 93.95% for CAS(ME)2, 93.42% for SMIC, 93.61% for SAMM and 93.79% for CASME II, surpassing the state-of-the-art methods across these datasets. These outcomes demonstrate the efficacy and robustness of the proposed pipeline, underscoring its potential for a range of micro-expression recognition uses.

References

[1] P. Zhang, X. Ben, R. Yan, C. Wu and C. Guo, "Micro-expression Recognition System," Optik (Stuttgart),vol. 127, no. 3, pp. 1395–1400, DOI: 10.1016/j.ijleo.2015.10.217, 2016.

[2] G. Zhao and X. Li, "Automatic Micro-expression Analysis: Open Challenges," Frontiers in Psychology,vol. 10, no. AUG, pp. 1–4, DOI: 10.3389/fpsyg.2019.01833, 2019.

[3] L. Zhou, X. Shao and Q. Mao, "A Survey of Micro-expression Recognition," Image and VisionComputing, vol. 105, p. 104043, DOI: 10.1016/j.imavis.2020.104043, 2021.

[4] Y. He, S. J. Wang, J. Li and M. H. Yap, "Spotting Macro-and Micro-expression Intervals in Long VideoSequences," Proc. of the 2020 15th IEEE Int. Conf. Autom. Face Gesture Recognition (FG 2020), pp. 742–748, DOI: 10.1109/FG47880.2020.00036, 2020.

[5] F. Qu, S. J. Wang,W. J. Yan, H. Li, S. Wu and X. Fu, "CAS(ME)2): A Database for Spontaneous Macro-expression and Micro-expression Spotting and Recognition," IEEE Transactions on Affective Computing, vol. 9, no. 4, pp. 424–436, DOI: 10.1109/TAFFC.2017.2654440, 2018.

[6] X. Li, P. Tomas, H. Xiaohua, Z. Guoying and P. Matti, "A Spontaneous Micro-expression Database:Inducement, Collection and Baseline," Proc. of the 2013 10th IEEE Int. Conf. and Workshops on Automatic Face and Gesture Recognition (FG), DOI: 10.1109/FG.2013.6553717, Shanghai, China, 2013.

[7] A. K. Davison et al., "SAMM : A Spontaneous Micro-facial Movement Dataset," IEEE Transactions onAffective Computing, vol. 9, no. 1, pp. 116–129, DOI: 10.1109/TAFFC.2016.2573832, 2016.

[8] W. J. Yan et al., "CASME II: An Improved Spontaneous Micro-expression Database and the BaselineEvaluation," PLoS One, vol. 9, no. 1, pp. 1–8, DOI: 10.1371/journal.pone.0086041, 2014.

[9] Y. Jiao, M. Jing, Y. Hu and K. Sun, "Research on a Micro-expression Recognition Algorithm Based on3D-CNN," Proc. of the 2021 3rd Int. Conf. Intell. Control. Meas. Signal Process. Intell. Oil Field (ICMSP 2021), no. Icmsp, pp. 221–225, DOI: 10.1109/ICMSP53480.2021.9513351, 2021.

[10] F. Guowen and L. Xi, "Micro-expression Recognition Based on Dual Branch Neural Network," Proc. ofthe 2023 Int. Conf. Artif. Intell. Comput. Inf. Technol. (AICIT 2023), no. 2020, pp. 2–5, DOI: 10.1109/AICIT59054.2023.10278020, 2023.

[11] Q. Zhou, S. Liu, Y. Wang and J. Wang, "Divided Block Multi-scale Convolutional Network for Micro-expression Recognition," Proc. of the 2022 1st Int. Conf. Cyber-Energy Syst. Intell. Energy (ICCSIE 2022), pp. 1–5, DOI: 10.1109/ICCSIE55183.2023.10175242, 2023.

[12] Z. Shang, J. Liu and X. Li, "Micro-expression Recognition Based on Spatio-temporal Capsule Network,"IEEE Access, vol. 11, no. January, pp. 13704–13713, DOI: 10.1109/ACCESS.2023.3242871, 2023.

[13] H. Tian, W. Gong, W. Li and Y. Qian, "PASTFNet: A Paralleled Attention Spatio-temporal FusionNetwork for Micro-expression Recognition," Medical and Biological Engineering and Computing, DOI: 10.1007/s11517-024-03041-y, 2024.

[14] A. J. Rakesh Kumar and B. Bhanu, "Relational Edge-node Graph Attention Network for Classificationof Micro-expressions," Proc. of the IEEE Computer Society Conf. on Computer Vision and Pattern Recognition, vol. 2023-June, pp. 5819–5828, DOI: 10.1109/CVPRW59228.2023.00618, 2023.

[15] F. Li, P. Nie, M. You, Z. Chen and G. Wang, "Triple-ATFME: Triple-branch Attention Fusion Networkfor Micro-expression Recognition," Arabian J. for Science and Eng., DOI: 10.1007/s13369-024-08973-z, 2024.

[16] H. Insan, S. S. Prasetiyowati and Y. Sibaroni, "SMOTE-LOF and Borderline-SMOTE Performance toOvercome Imbalanced Data and Outliers on Classification," Proc. of the 2023 3rd Int. Conf. Intell. Cybern. Technol. Appl., pp. 136–141, DOI: 10.1109/icicyta60173.2023.10428902, 2024.

[17] A. Sagoolmuang, "Power-weighted kNN Classification for Handling Class Imbalanced Problem," Proc.of the 2021 2nd Int. Conf. Big Data Anal. Pract. (IBDAP 2021), pp. 42–47, DOI: 10.1109/IBDAP52511.2021.9552164, 2021.

[18] B. Irawan, N. P. Utama, R. Munir and A. Purwarianti, "Improving the Accuracy of Facial Micro- expression Rrecognition: Spatio-temporal Deep Learning with Enhanced Data Augmentation and Class Balancing," Interdisciplinary Journal of Information, Knowledge, and Management, vol. 19, pp. 1–15, DOI: 10.28945/5386, 2024.

[19] Y. Zhou, H. Chen, J. Li, Y. Wu, J. Wu and L. Chen, "ST-Attn: Spatial-," Proc. of the IEEE Int. Conf.Data Min. Work. (ICDMW), vol. 2019-Novem, no. November, pp. 609–614, DOI: 10.1109/ICDMW.2019.00092, 2019.

[20] Y. S. Gan, S. E. Lien, Y. C. Chiang and S. T. Liong, "LAENet for Micro-expression Recognition,"Visual Computer, vol. 40, no. 2, pp. 585–599, DOI: 10.1007/s00371-023-02803-3, 2024.

[21] P. Sharma, S. Coleman, P. Yogarajah, L. Taggart and P. Samarasinghe, "Magnifying Spontaneous FacialMicro Expressions for Improved Recognition," Proc. of the Int. Conf. Pattern Recognit., pp. 7930–7936, DOI: 10.1109/ICPR48806.2021.9412585, 2020.

[22] M. Verma, S. K. Vipparthi, G. Singh and S. Murala, "LEARNet: Dynamic Imaging Network for MicroExpression Recognition," IEEE Transactions on Image Processing, vol. 29, no. c, pp. 1618–1627, DOI: 10.1109/TIP.2019.2912358, 2020.

[23] P. Gupta, "MERASTC: Micro-expression Recognition Using Effective Feature Encodings and 2DConvolutional Neural Network," IEEE Transactions on Affective Computing, vol. 14, no. 2, pp. 1431–1441, DOI: 10.1109/TAFFC.2021.3061967, 2023.

[24] S. Thuseethan, S. Rajasegarar and J. Yearwood, "Deep3DCANN : A Deep 3DCNN-ANN Framework forSpontaneous Micro-expression Recognition," Information Sciences (Ny), vol. 630, no. November 2022, pp. 341–355, DOI: 10.1016/j.ins.2022.11.113, 2023.

[25] L. Cai, H. Li, W. Dong and H. Fang, "Micro-expression Recognition Using 3D DenseNet Fused Squeeze-and-excitation Networks," Applied Soft Computing, vol. 119, p. 108594, DOI: 10.1016/j.asoc.2022.108594, 2022.

[26] W. S. P. Bayu and A. Setyanto, "3D CNN for Micro Expression Detection," Proc. of the 5th Int. Conf.Inf. Commun. Technol. A New W. to Make AI Useful Everyone New Norm. Era (ICOIACT 2022), pp. 397–401, DOI: 10.1109/ICOIACT55506.2022.9972194, 2022.

[27] B. Chen, K. H. Liu, Y. Xu, Q. Q. Wu and J. F. Yao, "Block Division Convolutional Network with ImplicitDeep Features Augmentation for Micro-expression Recognition," IEEE Transactions onMultimedia, vol. 25, pp. 1345–1358, DOI: 10.1109/TMM.2022.3141616, 2023.

[28] M. Verma, P. Lubal, S. K. Vipparthi and M. Abdel-Mottaleb, "RNAS-MER: A Refined NeuralArchitecture Search with Hybrid Spatiotemporal Operations for Micro-expression Recognition," Proc. of the 2023 IEEE Winter Conf. Appl. Comput. Vision (WACV 2023), pp. 4759–4768, DOI: 10.1109/WACV56688.2023.00475, 2023.

[29] Z. Zhai and J. Zhao, "Feature Representation Learning with Adaptive Displacement Generation andTransformer Fusion for Micro-expression Recognition," Proc. of the 2023 IEEE/CVF Conf. Comput. Vis. Pattern Recognit., pp. 22086–22095, DOI: 10.1109/CVPR52729.2023.02115, 2023.

[30] Z. Li, Y. Zhang, H. Xing and K.-L. Chan, "Facial Micro-expression Recognition Using Double‑Stream3D Convolutional Neural Network with Domain Adaptation," Sensors, vol. 23, no. 7, DOI: 10.3390/s23073577, 2023.

[31] Y. Wang, H. U. Shi and R. Wang, "Action Decouple Multi-tasking for Micro-expression Recognition,"IEEE Access, vol. 11, no. June, pp. 82978–82988, DOI: 10.1109/ACCESS.2023.3301950, 2023.