浏览全部资源
扫码关注微信
Published： 16 May 2024 ，
移动端阅览
李博凯， 吴从中， 项柏杨， 臧怀娟， 任永生， 詹曙. 2024. 微表情峰值帧定位引导的分类算法. 中国图象图形学报， 29(05):1447-1459
Li Bokai， Wu Congzhong， Xiang Baiyang， Zang Huaijuan， Ren Yongsheng， Zhan Shu. 2024. Apex frame spotting and recognition of micro-expression by optical flow. Journal of Image and Graphics， 29(05):1447-1459
目的
2
微表情是人在外界信息和刺激下做出的无意识面部动作，是判断受试人情绪和行为的重要佐证，在社会安全、商业谈判和心理辅导等领域都有着广泛的应用。微表情不同于一般的表情，分类与定位较为困难。针对这种情况，提出了一种基于光流窗口的双分支微表情定位网络（dual-branch optical flow spotting network，DFSN）和一种利用峰值帧光流信息的微表情分类网络，以识别视频中的微表情。
方法
2
在定位任务中，首先提取面部图像，选择光流窗口大小和位置，计算面部光流并进行预处理；接下来输入双分支网络中进行两次分类，分别针对有无微表情和在有微表情前提下微表情所处阶段分类，并结合两个损失函数抑制过拟合；最后绘制出微表情强度曲线，曲线峰值所处位置即为所求微表情峰值帧。在分类任务中，选取视频起始帧和定位网络取得的峰值帧作为光流窗口，并利用欧拉运动放大算法（Eulerian motion magnification，EMM）放大微表情，最后采用峰值帧光流信息分类微表情视频。
结果
2
微表情定位网络分别在CASME II（Chinese Academy of Sciences Micro-expression Database II）数据集和CASME数据集上按照使用留一被试交叉验证法进行了实验，与目前最好的定位方法比较，此网络在CASME II上获得了最低的NMAE（normalized mean absolute error）值0.101 7，比Optical flow + UPC方法提高了9%。在CASME上获得的NMAE值为0.137 8，在此数据集上为次优定位方法。在定位网络得到的峰值基础上，分类网络在CASME II上取得了89.79%的准确率，在CASME上取得了66.06%的准确率。若采用数据集标注的峰值，分类网络在CASME II上取得了91.83%的准确率，在CASME上取得了76.96%的准确率。
结论
2
提出的微表情定位网络可以有效定位视频中微表情峰值帧的位置，帮助后续网络进行分类，微表情分类网络可以有效区分不同种类的微表情视频。
Objective
2
Micro-expressions are unconscious facial actions made by people under external information and stimulation. These expressions are crucial proofs to judge people’s emotions and thoughts. Micro-expressions are widely used in the fields of social security， business negotiation， and psychological counseling. This type of expression is different from the general macro-expression and demonstrates characteristics of short duration， low expression intensity， and fast change speed. Therefore， compared with macro-expressions， micro-expressions are more difficult to recognize and locate. Before the emergence of deep learning， researchers mostly used the traditional hand-crafted method， which utilizes the artificially designed micro-expression extractors and complex parameter adjustment processes and algorithms to extract features. Some excellent algorithms can achieve competitive results， such as local binary pattern-three orthogonal plane and main directional mean optical flow （MDMO）. However， these algorithms mostly only extract shallow features， and improving their accuracy is difficult. With the development of machine learning in the field of computer vision， the research method of micro-expression based on deep learning has immediately become the mainstream. This method generally uses convolutional neural network to extract and classify the image or video features. The accuracy of micro-expression identification is markedly improved due to its powerful feature extraction and learning capability. However， the spotting and classification of micro-expressions are still difficult tasks due to the subtle characteristics of micro-expressions and the difficulty of extracting effective features. Therefore， this paper proposes a dual-branch optical flow spotting network based on optical flow window， which can promote the solution of these problems.
Method
2
First， the size of the optical flow window is selected in accordance with the number of video frames， and three frames at both ends of the window are taken to stabilize the optical flow intensity. Dlib library is used to detect faces， and Farneback method is used to extract facial optical flow features and preprocess the optical flow image. The image size is finally converted into 224 × 224 pixels.The dual-branch network is then inputted for two classifications to address the presence or absence of micro-expression and the rising or falling state of micro-expression. The twice classification should be judged in accordance with the same characteristics. Therefore， the same network backbone is used， and then the branches are utilized to process the characteristics， thereby focusing on different directions. Combining two loss functions can suppress the overfitting of the network， complete classification， and improve the network performance. Finally， the micro-expression state in the video window is obtained by sliding the window， and the intensity curve is drawn. Multiple windows are selected for positioning due to the different durations of micro-expression， and the highest point among them is taken as the apex frame. The classification network is different from the location network in two aspects. First， the front end of the window is the second to the fourth frame of the video and the back end uses the micro-expression part of the video. Second， Euler motion magnification is used to process video. This method can amplify facial motion and improve expression intensity but will destroy some optical flow features； thus， the method is not used in the positioning network. When classifying videos， the apex frame of the positioning network is taken as the center， and the five surrounding positions are selected as the input of the classification network. The classification network uses the uncomplicated network structure and obtains good results， proving the importance of apex frame spotting.
Result
2
The micro-expression spotting network is based on leave-one-subject-out cross-validation method on the Chinese Academy of Sciences Micro-expression Database II （CASME II） and the Chinese Academy of Sciences Micro-expression Database （CASME）， which is the most commonly used validation method in the current micro-expression identification research. Compared with the current best spotting method， the lowest normalized mean absolute error （NMAE） value of 0.101 7 is obtained on the CASME II， which is 9% lower than the current best spotting method. The NMAE value obtained on the CASME is 0.137 8， which is currently the second lowest number. Using this micro-expression spotting network， the classification network achieved 89.79% accuracy of three categories （positive， negative， and surprise） in the micro-expression classification experiment of CASME II and 66.06% accuracy of four categories （disgust， tense， repression， and surprise） in the micro-expression classification experiment of CASME. Using the apex frame in dataset， the classification network achieved 91.83% and 76.96% accuracy on CASME II and CASME， respectively.
Conclusion
2
The proposed micro-expression spotting network can effectively locate the position of the apex frame in the video and then extract its effective micro-expression information. Extensive experimental evaluation proved that the spotting network has good spotting effect. The subsequent classification network shows that the extraction of effective micro-expression information such as an apex frame can significantly help the network in classifying micro-expressions. Overall， the proposed micro-expression spotting network can substantially improve the accuracy of micro-expression recognition.
微表情定位情感计算峰值帧微表情分类图像识别深度学习
micro-expression spottingaffective computingapex framemicro-expression classificationimage recognitiondeep learning
Ekman P. 2008. Telling Lies： Clues to Deceit in the Marketplace， Politics， and Marriage. Deng B C， trans. Beijing： SDX Joint Publishing Company
Ekman P. 2008. 说谎： 揭穿商业、政治与婚姻中的骗局. 邓伯宸， 译. 北京： 生活·读书·新知三联书店
Farnebäck G. 2003. Two-frame motion estimation based on polynomial expansion//Proceedings of the 13th Scandinavian Conference on Image Analysis. Halmstad， Sweden： Springer： 363-370 ［DOI： 10.1007/3-540-45103-X_50http://dx.doi.org/10.1007/3-540-45103-X_50］
Gan Y S， Liong S T， Yau W C， Huang Y C and Tan L K. 2019. OFF-ApexNet on micro-expression recognition system. Signal Processing： Image Communication， 74： 129-139 ［DOI： 10.48550/arXiv.1805.08699http://dx.doi.org/10.48550/arXiv.1805.08699］
Happy S L and Routray A. 2019. Fuzzy histogram of optical flow orientations for micro-expression recognition. IEEE Transactions on Affective Computing， 10（3）： 394-406 ［DOI： 10.1109/TAFFC.2017.2723386http://dx.doi.org/10.1109/TAFFC.2017.2723386］
Huang X H， Zhao G Y， Hong X P， Zheng W M and Pietikäinen M. 2016. Spontaneous facial micro-expression analysis using spatiotemporal completed local quantized patterns. Neurocomputing， 175： 564–578 ［DOI： 10.1016/j.neucom.2015.10.096http://dx.doi.org/10.1016/j.neucom.2015.10.096］
Huang X H and Zhao G Y. 2017. Spontaneous facial micro-expression analysis using spatiotemporal local radon-based binary pattern//Proceedings of 2017 International Conference on the Frontiers and Advances in Data Science （FADS）. Xi’an， China： IEEE： 159-164 ［DOI： 10.1109/FADS.2017.8253219http://dx.doi.org/10.1109/FADS.2017.8253219］
Khor H Q， See J， Liong S T， Phan R C W and Lin W Y. 2019. Dual-stream shallow networks for facial micro-expression recognition//Proceedings of 2019 IEEE International Conference on Image Processing （ICIP）. Taipei， China： IEEE： 36-40 ［DOI： 10.1109/ICIP.2019.8802965http://dx.doi.org/10.1109/ICIP.2019.8802965］
Kumar A J R and Bhanu B. 2021. Micro-expression classification based on landmark relations with graph attention convolutional network//Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville， USA： IEEE： 1511-1520 ［DOI： 10.1109/CVPRW53098.2021.00167http://dx.doi.org/10.1109/CVPRW53098.2021.00167］
Lei L， Chen T， Li S G and Li J F. 2021. Micro-expression recognition based on facial graph representation learning and facial action unit fusion//Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops （CVPRW）. Nashville， USA： IEEE： 1571-1580 ［DOI： 10.1109/CVPRW53098.2021.00173http://dx.doi.org/10.1109/CVPRW53098.2021.00173］
Lei L， Li J F， Chen T and Li S G. 2020. A novel Graph-TCN with a graph structured representation for micro-expression recognition//Proceedings of the 28th ACM International Conference on Multimedia. Seattle， USA： ACM： 2237-2245 ［DOI： 10.1145/3394171.3413714http://dx.doi.org/10.1145/3394171.3413714］
Li X H， Yu J， Zhan S. 2016. Spontaneous facial micro-expression detection based on deep learning//Proceedings of the 13th IEEE International Conference on Signal ProcessingICSP2016. Chengdu， China： IEEE： 1130-1134 ［DOI： 10.1109/ICSP.2016.7878004http://dx.doi.org/10.1109/ICSP.2016.7878004］
Li Y T， Huang X H and Zhao G Y. 2018. Can micro-expression be recognized based on single apex frame//Proceedings of the 25th IEEE International Conference on Image Processing （ICIP）. Athens， Greece： IEEE： 3094-3098 ［DOI： 10.1109/ICIP.2018.8451376http://dx.doi.org/10.1109/ICIP.2018.8451376］
Li Y T， Huang X H and Zhao G Y. 2021a. Micro-expression action unit detection with spatial and channel attention. Neurocomputing， 436： 221-231 ［DOI： 10.1016/j.neucom.2021.01.032http://dx.doi.org/10.1016/j.neucom.2021.01.032］
Li Y T， Huang X H and Zhao G Y. 2021b. Joint local and global information learning with single apex frame detection for micro-expression recognition. IEEE Transactions on Image Processing， 30： 249-263 ［DOI： 10.1109/TIP.2020.3035042http://dx.doi.org/10.1109/TIP.2020.3035042］
Liong S T， See J， Wong K， Le Ngo A C， Oh Y H and Phan R. 2015. Automatic apex frame spotting in micro-expression database//Proceedings of the 3rd IAPR Asian Conference on Pattern Recognition （ACPR）. Kuala Lumpur， Malaysia： IEEE： 665-669 ［DOI： 10.1109/ACPR.2015.7486586http://dx.doi.org/10.1109/ACPR.2015.7486586］
Liong S T， See J， Wong K and Phan R C W. 2018. Less is more： microexpression recognition from video using apex frame. Signal Processing： Image Communication， 62： 82-92 ［DOI： 10.1016/j.image.2017.11.006http://dx.doi.org/10.1016/j.image.2017.11.006］
Liu D Z， Liang Z Y and Sun Y. 2021. Micro-expression recognition method based on spatial attention mechanism and optical flow features. Journal of Computer-Aided Design and Computer Graphics， 33（10）： 1541-1552
刘德志， 梁正友， 孙宇. 2021. 结合空间注意力机制与光流特征的微表情识别方法. 计算机辅助设计与图形学学报， 33（10）： 1541-1552 ［DOI： 10.3724/SP.J.1089.2021.18569http://dx.doi.org/10.3724/SP.J.1089.2021.18569］
Liu Y J， Zhang J K， Yan W J， Wang S J， Zhao G Y and Fu X L. 2016. A main directional mean optical flow feature for spontaneous micro-expression recognition. IEEE Transactions on Affective Computing， 7（4）： 299-310 ［DOI： 10.1109/taffc.2015.2485205http://dx.doi.org/10.1109/taffc.2015.2485205］
Lo L， Xie H X， Shuai H H and Cheng W H. 2020. MER-GCN： micro-expression recognition based on relation modeling with graph convolutional networks//Proceedings of 2020 IEEE Conference on Multimedia Information Processing and Retrieval （MIPR）. Shenzhen， China： IEEE： 79-84 ［DOI： 10.1109/MIPR49039.2020.00023http://dx.doi.org/10.1109/MIPR49039.2020.00023］
Ma H Y， An G Y， Wu S J and Yang F. 2017. A region histogram of oriented optical flow （RHOOF） feature for apex frame spotting in micro-expression//2017 International Symposium on Intelligent Signal Processing and Communication Systems （ISPACS）. Xiamen， China： IEEE： 281-286 ［DOI： 10.1109/ISPACS.2017.8266489http://dx.doi.org/10.1109/ISPACS.2017.8266489］
Min R P， Li Y F， Huang Y， Yang J Y and Zhong B J. 2022. Visualized all-scale shape representation and recognition. Journal of Image and Graphics， 27（2）： 628-641
闵睿朋， 李一凡， 黄瑶， 杨剑宇， 钟宝江. 2022. 形状的全尺度可视化表示与识别. 中国图象图形学报， 27（2）： 628-641 ［DOI： 10.11834/jig.200693http://dx.doi.org/10.11834/jig.200693］
Niu M Y， Tao J H， Li Y， Huang J and Lian Z. 2019. Discriminative video representation with temporal order for micro-expression recognition//Proceedings of 2019 IEEE International Conference on Acoustics， Speech and Signal Processing （ICASSP）. Brighton， UK： IEEE： 2112-2116 ［DOI： 10.1109/ICASSP.2019.8682295http://dx.doi.org/10.1109/ICASSP.2019.8682295］
Niu R H， Yang J， Xing L X and Wu R B. 2021. Micro-expression recognition algorithm based on convolutional block attention module and dual path networks. Journal of Computer Applications， 41（9）： 2552-2559
牛瑞华， 杨俊， 邢斓馨， 吴仁彪. 2021. 基于卷积注意力模块和双通道网络的微表情识别算法. 计算机应用， 41（9）： 2552-2559 ［DOI： 10.11772/j.issn.1001-9081.2020111743http://dx.doi.org/10.11772/j.issn.1001-9081.2020111743］
Patel D， Zhao G Y and Pietikäinen M. 2015. Spatiotemporal integration of optical flow vectors for micro-expression detection//Proceedings of the 16th International Conference on Advanced Concepts for Intelligent Vision Systems. Catania， Italy： Springer： 369-380 ［DOI： 10.1007/978-3-319-25903-1_32http://dx.doi.org/10.1007/978-3-319-25903-1_32］
Peng M， Wang C Y， Chen T， Liu G Y and Fu X L. 2017. Dual temporal scale convolutional neural network for micro-expression recognition. Frontiers in Psychology， 8： #1745 ［DOI： 10.3389/fpsyg.2017.01745http://dx.doi.org/10.3389/fpsyg.2017.01745］
Peng M， Wu Z， Zhang Z H and Chen T. 2018. From macro to micro expression recognition： deep learning on small datasets using transfer learning//Proceedings of the 13th IEEE International Conference on Automatic Face and Gesture Recognition （FG 2018）. Xi’an， China： IEEE： 657-661 ［DOI： 10.1109/FG.2018.00103http://dx.doi.org/10.1109/FG.2018.00103］
Pfister T， Li X B， Zhao G Y and Pietikäinen M. 2011. Recognising spontaneous facial micro-expressions// Proceedings of 2011 International Conference on Computer Vision. Barcelona， Spain： IEEE： 1449-1456 ［DOI： 10.1109/ICCV.2011.6126401http://dx.doi.org/10.1109/ICCV.2011.6126401］
Shang Z Y， Liu J and Li X F. 2023. Micro-expression recognition based on spatio-temporal capsule network. IEEE Access， 11： 13704-13713 ［DOI： 10.1109/ACCESS.2023.3242871http://dx.doi.org/10.1109/ACCESS.2023.3242871］
She W X， Liu B， Tao J H， Zhang H and Lyu Z. 2021. Aggregation of motion features of multiple paths for micro-expression recognition. Journal of Computer-Aided Design and Computer Graphics， 33（9）： 1457-1465
佘文祥， 刘斌， 陶建华， 张昊， 吕钊. 2021. 多通道运动特征融合的微表情识别方法. 计算机辅助设计与图形学学报， 33（9）： 1457-1465 ［DOI： 10.3724/SP.J.1089.2021.18725http://dx.doi.org/10.3724/SP.J.1089.2021.18725］
Song B L， Li K， Zong Y， Zhu J， Zheng W M， Shi J G and Zhao L. 2019. Recognizing spontaneous micro-expression using a three-stream convolutional neural network. IEEE Access， 7： 184537-184551 ［DOI： 10.1109/ACCESS.2019.2960629http://dx.doi.org/10.1109/ACCESS.2019.2960629］
Wang Q L， Wu B G， Zhu P F， Li P H， Zuo W M and Hu Q H. 2020. ECA-Net： efficient channel attention for deep convolutional neural networks//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition （CVPR）. Seattle， USA： IEEE： 11531-11539 ［DOI： 10.1109/CVPR42600.2020.01155http://dx.doi.org/10.1109/CVPR42600.2020.01155］
Wang S J， Yan W J， Li X B， Zhao G Y and Fu X L. 2014. Micro-expression recognition using dynamic textures on tensor independent color space//Proceedings of the 22nd International Conference on Pattern Recognition. Stockholm， Sweden： IEEE： 4678-4683 ［DOI： 10.1109/ICPR.2014.800http://dx.doi.org/10.1109/ICPR.2014.800］
Wang Y D， See J， Phan R C W and Oh Y H. 2015. LBP with six intersection points： reducing redundant information in LBP-TOP for micro-expression recognition//Proceedings of the 12th Asian Conference on Computer Vision. Singapore， Singapore： Springer： 525-537 ［DOI： 10.1007/978-3-319-16865-4_34http://dx.doi.org/10.1007/978-3-319-16865-4_34］
Wu H Y， Rubinstein M， Shih E， Guttag J， Durand F and Freeman W. 2012. Eulerian video magnification for revealing subtle changes in the world. ACM Transactions on Graphics， 31（4）： #65 ［DOI： 10.1145/2185520.2185561http://dx.doi.org/10.1145/2185520.2185561］
Xia Z Q， Hong X P， Gao X Y， Feng X Y and Zhao G Y. 2020. Spatiotemporal recurrent convolutional networks for recognizing spontaneous micro-expressions. IEEE Transactions on Multimedia， 22（3）： 626-640 ［DOI： 10.1109/TMM.2019.2931351http://dx.doi.org/10.1109/TMM.2019.2931351］
Xie H X， Lo L， Shuai H H and Cheng W H. 2020. AU-assisted graph attention convolutional network for micro-expression recognition//Proceedings of the 28th ACM International Conference on Multimedia. Seattle， USA： ACM： 2871-2880 ［DOI： 10.1145/3394171.3414012http://dx.doi.org/10.1145/3394171.3414012］
Xu F， Zhang J P and Wang J Z. 2017. Microexpression identification and categorization using a facial dynamics map. IEEE Transactions on Affective Computing， 8（2）： 254-267 ［DOI： 10.1109/TAFFC.2016.2518162http://dx.doi.org/10.1109/TAFFC.2016.2518162］
Yan W J， Li X B， Wang S J， Zhao G Y， Liu Y J， Chen Y H and Fu X L. 2014a. CASME II： an improved spontaneous micro-expression database and the baseline evaluation. PLoS One， 9（1）： #86041 ［DOI： 10.1371/journal.pone.0086041http://dx.doi.org/10.1371/journal.pone.0086041］
Yan W J， Wang S J， Chen Y H， Zhao G Y and Fu X L. 2014b. Quantifying micro-expressions with constraint local model and local binary pattern//Computer Vision — ECCV 2014 Workshops. Zurich， Switzerland： Springer： 296-305 ［DOI： 10.1007/978-3-319-16178-5_20http://dx.doi.org/10.1007/978-3-319-16178-5_20］
Yan W J， Wu Q， Liu Y J， Wang S J and Fu X L. 2013. CASME database： a dataset of spontaneous micro-expressions collected from neutralized faces//Proceedings of the 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition. Shanghai， China： IEEE： 1-7 ［DOI： 10.1109/FG.2013.6553799http://dx.doi.org/10.1109/FG.2013.6553799］
Yang Z M and Song W. 2023. Selecting and fusing coarse-and-fine granularity features for fine-grained image recognition. Journal of Image and Graphics， 28（7）： 2081-2092
阳治民， 宋威. 2023. 选择并融合粗细粒度特征的细粒度图像识别. 中国图象图形学报， 28（7）： 2081-2092 ［DOI： 10.11834/jig.220052http://dx.doi.org/10.11834/jig.220052］
Zhao S R， Tang H Y， Liu S F， Zhang Y S， Wang H， Xu T， Chen E H and Guan C T. 2022. ME-PLAN： a deep prototypical learning with local attention network for dynamic micro-expression recognition. Neural Networks， 153： 427-443 ［DOI： 10.1016/j.neunet.2022.06.024http://dx.doi.org/10.1016/j.neunet.2022.06.024］
Zhao S R， Tao H Q， Zhang Y S， Xu T， Zhang K， Hao Z K and Chen E H. 2021. A two-stage 3D CNN based learning method for spontaneous micro-expression recognition. Neurocomputing， 448： 276-289 ［DOI： 10.1016/j.neucom.2021.03.058http://dx.doi.org/10.1016/j.neucom.2021.03.058］
Zhi R C， Hu J and Wan F. 2022. Micro-expression recognition with supervised contrastive learning. Pattern Recognition Letters， 163： 25-31 ［DOI： 10.1016/j.patrec.2022.09.006http://dx.doi.org/10.1016/j.patrec.2022.09.006］
Zhou L， Shao X Y and Mao Q R. 2021. A survey of micro-expression recognition. Image and Vision Computing， 105： #104043 ［DOI： 10.1016/j.imavis.2020.104043http://dx.doi.org/10.1016/j.imavis.2020.104043］
相关作者
相关机构
京公网安备11010802024621