浏览全部资源
扫码关注微信
Published： 16 July 2024 ，
移动端阅览
成娟， 殷辰楚， 宋仁成， 付静， 刘羽. 2024. 非规律运动伪迹干扰鲁棒的人脸视频心率检测. 中国图象图形学报， 29(07):2024-2034
Cheng Juan， Yin Chenchu， Song Rencheng， Fu Jing， Liu Yu. 2024. Facial video-based heart rate measurement against irregular motion artifacts. Journal of Image and Graphics， 29(07):2024-2034
目的
2
基于远程光电容积脉搏波描记法（remote photoplethysmograph，rPPG）的非接触人脸视频心率检测广泛应用于移动健康监护领域，由于其携带的生理参数信息幅值微弱，容易受到运动伪迹干扰。据此，提出了一种结合非负矩阵分解（nonnegative matrix factorization，NMF）和独立向量分析（independent vector analysis，IVA）的非规律运动伪迹去除的视频心率检测方法，记为NMF-IVA。
方法
2
首先，将面部感兴趣区域（region of interest，RoI）分为多个子区域（sub RoIs，SRoIs），利用平均光照强度、光照强度变化、信噪比这3个指标筛选出3个最优质的SRoIs，并获取每个SRoI的绿色通道时间序列。其次，将3个绿色通道时间序列去趋势、带通滤波后送入NMF-IVA进行盲源分离。然后，对分离后的源信号进行功率谱密度分析，并且将峰值信噪比最高且主频落在心率感兴趣范围内的源信号确定为血容量脉冲（blood volume pulse，BVP）信号。最后，将BVP信号的主频确定为所测量心率的主频，从而计算出心率值。
结果
2
实验在2个公开数据集UBFC-RPPG和UBFC-PHYS，及1个真实场景自采数据集上与最相关的7种典型的rPPG方法进行比较，在UBFC-RPPG数据集上，相比于性能第2的单通道滤波（single channel filtering，SCF）方法，均方根误差提升了1.39 bpm（beat per minute）、平均绝对误差提升了1.25 bpm、皮尔逊相关系数提升了0.02；在UBFC-PHYS数据集上的T2情况下，其性能提升最为显著，相比于性能第2的独立向量分析（IVA）方法，均方根误差提升了16.42 bpm、平均绝对误差提升了9.91 bpm、皮尔逊相关系数提升了0.64；在自采数据集上，除了低于深度学习方法性能之外，所提NMF-IVA方法在传统方法中取得了最好的结果。
结论
2
所提NMF-IVA方法对规律信号提取具有敏感性，即便是在头部存在剧烈非规律运动情况下，相比于传统方法亦能取得最优结果，该结果能够媲美基于深度学习的方法。
Objective
2
Heart rate （HR） is one of the most important physiological parameters that can reflect the physical and mental status of individuals. Various methods have been developed to estimate HR values using contact and noncontact sensors. The advantage of noncontact methods is that they provide a more comfortable and unobtrusive way to estimate HR and avoid discomfort or skin allergy caused by conventional contact methods. The pulse-induced subtle color variations of facial skins can be measured from consumer-level cameras. Thus， camera-based non-contact HR detection technology， also called remote photoplethysmograph （rPPG）， has been widely used in the fields of mobile health monitoring， driving safety， and emotion awareness. The principle of camera-based rPPG measurement is similar to that of traditional PPG measurement， that is， the pulsatile blood propagating in cardiovascular systems changes blood volumes in microvascular tissue beds beneath skins with each heartbeat， thus producing a fluctuation. However， such technology is susceptible to motion artifacts due to weak amplitudes of the physiological parameter information it carries. For instance， subjects’ heads may move involuntarily during interviews， presentations， and other socially stressful situations， thus degrading rPPG-based HR detection performance. Accordingly， this paper proposes a novel motion-robust rPPG method that combines nonnegative matrix factorization （NMF） and independent vector analysis （IVA）， termed as NMF-IVA， to remove irregular motion artifacts.
Method
2
First， the whole facial region of interest （RoI） is divided into several sub RoIs （SRoIs）， among which three optimal SRoIs are selected based on three indicators： average light intensity， light intensity variation of a certain SRoI， and signal-to-noise ratio （SNR） of the green-channel signal derived from the SRoI. Afterwards， three green-channel time series are derived from the corresponding three optimal SRoIs. Second， the three channels of time series are detrended， bandpass filtered， and then sent to the proposed NMF-IVA as input. After the NMF-IVA operation， three source signals are extracted and then processed by power spectral density analysis. The one with the highest peak SNR and the corresponding dominant frequency falling within the interested HR range will be identified as the blood volume pulse （BVP） signal， whose dominant frequency is identified as that of the estimated HR.
Result
2
We compare the proposed NMF-IVA method with seven typical rPPG methods on two publicly available datasets（UBFC-RPPG and UBFC-PHYS） as well as one in-house dataset. On the UBFC-RPPG dataset， compared with the second-best performance of the single channel filtering （SCF） method， the proposed NMF-IVA achieves better performance， with an improved root mean square error （RMSE） of HR measurement by 1.39 beat per minute （bpm）， an improved mean absolute error （MAE） by 1.25 bpm， and a higher Pearson’s correlation coefficient （PCC） by 0.02. Although both the MAE and the RMSE achieved by the proposed NMF-IVA method are lower than those of deep learning-based methods， the PCC of the NMF-IVA is comparable to that of deep learning-based ones， which demonstrates the effectiveness of the proposed NMF-IVA method. As for the UBFC-PHYS dataset when compared with traditional rPPG methods， during the T1 condition， the performance of the proposed NMF-IVA method is better than that of the second-best SCF method， with an improved RMSE by 6.45 bpm， an improved MAE by 2.53 bpm， and a higher PCC by 0.18. When compared with deep learning-based ones， the proposed NMF-IVA method achieves the second-best performance. The performance improvement of the proposed NMF-IVA is most noticeable during the T2 condition on the UBFC-PHYS dataset. Specifically， when compared with the second-best performance of IVA， the above three metrices are improved by 16.42 bpm， 9.91 bpm， and 0.64， respectively. As for the UBFC-PHYS dataset， when during the T3 condition， the best performance is still achieved by the proposed NMF-IVA method. When compared with the second-best performance of the independent component analysis method， the corresponding three metrices are improved by 8.54 bpm， 6.14 bpm， and 0.37， respectively. The performance of the proposed NMF-IVA method can be comparable to that of deep learning-based ones both in T2 and T3 conditions. As for the in-house dataset， the proposed NMF-IVA method achieves better performance compared with the traditional methods， except for deep learning-based methods.
Conclusion
2
The proposed NMF-IVA method achieves the best results on all the three datasets when compared with traditional rPPG methods， and the performance improvement is most noticeable during irregular motion artifact conditions involving head motions with large amplitudes. However， the performance of the proposed NMF-IVA method is slightly poorer than that of deep learning-based methods possibly because deep learning technology has excellent abilities in learning and extracting effective features. However， sufficient training samples and generalization should be considered when adopting deep learning-based methods. In addition， before the high-quality BVP source is derived， upsampling is employed， which leads to a relatively large time consumption. In the future， the HR estimation performance and the upsampling rate should be traded off. The proposed NMF-IVA method has advantages in extracting regular signals. Thus， our study can provide a new solution for promoting the practical application ability of rPPG technology.
远程光电容积脉搏波描记法（rPPG）非接触式心率检测盲源分离（BSS）非负矩阵分解（NMF）独立向量分析 （IVA）
remote photoplethysmograph （rPPG）non-contact heart rate measurementblind source separation（BSS）nonnegative matrix factorization （NMF）independent vector analysis （IVA）
Algumaei A， Azam M， Najar F and Bouguila N. 2023. Bounded multivariate generalized Gaussian mixture model using ICA and IVA. Pattern Analysis and Applications， 26（3）： 1223-1252 ［DOI： 10.1007/s10044-023-01148-whttp://dx.doi.org/10.1007/s10044-023-01148-w］
Biswas D， Simões-Capela N， Van Hoof C and van Helleputte N. 2019. Heart rate estimation from wrist-worn photoplethysmography： a review. IEEE Sensors Journal， 19（16）： 6560-6570 ［DOI： 10.1109/JSEN.2019.2914166http://dx.doi.org/10.1109/JSEN.2019.2914166］
Bobbia S， Macwan R， Benezeth Y， Mansouri A and Dubois J. 2019. Unsupervised skin tissue segmentation for remote photoplethysmography. Pattern Recognition Letters， 124： 82-90 ［DOI： 10.1016/j.patrec.2017.10.017http://dx.doi.org/10.1016/j.patrec.2017.10.017］
Chen J， Chang Z Q， Qiu Q， Li X B， Sapiro G， Bronstein A and Pietikäinen M. 2016. RealSense = real heart rate： illumination invariant heart rate estimation from videos//Proceedings of the 6th International Conference on Image Processing Theory， Tools and Applications （IPTA）. Oulu， Finland： IEEE： 1-6 ［DOI： 10.1109/IPTA.2016.7820970http://dx.doi.org/10.1109/IPTA.2016.7820970］
Chen X， Cheng J， Song R C， Liu Y， Ward R and Wang Z J. 2019. Video-based heart rate measurement： recent advances and future prospects. IEEE Transactions on Instrumentation and Measurement， 68（10）： 3600-3615 ［DOI： 10.1109/TIM.2018.2879706http://dx.doi.org/10.1109/TIM.2018.2879706］
Cheng J， Wang P， Song R C， Liu Y， Li C， Liu Y and Chen X. 2021. Remote heart rate measurement from near-infrared videos based on joint blind source separation with delay-coordinate transformation. IEEE Transactions on Instrumentation and Measurement， 70： #5005313 ［DOI： 10.1109/TIM.2020.3041083http://dx.doi.org/10.1109/TIM.2020.3041083］
De Haan G and Jeanne V. 2013. Robust pulse rate from chrominance-based rPPG. IEEE Transactions on Biomedical Engineering， 60（10）： 2878-2886 ［DOI： 10.1109/TBME.2013.2266196http://dx.doi.org/10.1109/TBME.2013.2266196］
den Brinker A C and Wang W J. 2022. Model-based camera-PPG： Pulse-rate monitoring in fitness//Wang W J and Wang X Y， eds. Contactless Vital Signs Monitoring. ［s.l.］： Academic Press： 51-78 ［DOI： 10.1016/B978-0-12-822281-2.00011-1http://dx.doi.org/10.1016/B978-0-12-822281-2.00011-1］
Hu M， Guo D， Jiang M X， Qian F， Wang X H and Ren F J. 2022. rPPG-based heart rate estimation using spatial-temporal attention network. IEEE Transactions on Cognitive and Developmental Systems， 14（4）： 1630-1641 ［DOI： 10.1109/TCDS.2021.3131197http://dx.doi.org/10.1109/TCDS.2021.3131197］
Kwon S， Kim J， Lee D and Park K. 2015. ROI analysis for remote photoplethysmography on facial video//Proceedings of the 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society. Milan， Italy： IEEE： 4938-4941 ［DOI： 10.1109/EMBC.2015.7319499.PMID：26737399http://dx.doi.org/10.1109/EMBC.2015.7319499.PMID：26737399］
Li B， Zhang P P， Peng J Y and Fu H. 2023. Non-contact PPG signal and heart rate estimation with multi-hierarchical convolutional network. Pattern Recognition， 139： #109421 ［DOI： 10.1016/j.patcog.2023.109421http://dx.doi.org/10.1016/j.patcog.2023.109421］
Liu X， Fromm J， Patel S and McDuff D. 2021a. Multi-task temporal shift attention networks for on-device contactless vitals measurement ［EB/OL］. ［2023-06-25］. https://arxiv.org/pdf/2006.03790v2.pdfhttps://arxiv.org/pdf/2006.03790v2.pdf
Liu X， Jiang Z H， Fromm J， Xu X H， Patel S and McDuff D. 2021b. MetaPhys： few-shot adaptation for non-contact physiological measurement//Proceedings of 2021 Conference on Health， Inference， and Learning. Virtual Event， USA： ACM： 154-163 ［DOI： 10.1145/3450439.3451870http://dx.doi.org/10.1145/3450439.3451870］
Liu X， Hill B， Jiang Z H， Patel S and McDuff D. 2023. EfficientPhys： enabling simple， fast and accurate camera-based cardiac measurement//Proceedings of 2023 IEEE/CVF Winter Conference on Applications of Computer Vision. Waikoloa， USA： IEEE： 4997-5006 ［DOI： 10.1109/WACV56688.2023.00498http://dx.doi.org/10.1109/WACV56688.2023.00498］
Niu X S， Han H and Shan S G. 2020. Remote photoplethysmography-based physiological measurement： a survey. Journal of Image and Graphics， 25（11）： 2321-2336
牛雪松， 韩琥， 山世光. 2020. 基于rPPG的生理指标测量方法综述. 中国图象图形学报， 25（11）： 2321-2336 ［DOI： 10.11834/jig.200341http://dx.doi.org/10.11834/jig.200341］
Poh M Z， McDuff D J and Picard R W. 2011. Advancements in noncontact， multiparameter physiological measurements using a webcam. IEEE Transactions on Biomedical Engineering， 58（1）： 7-11 ［DOI： 10.1109/TBME.2010.2086456http://dx.doi.org/10.1109/TBME.2010.2086456］
Ruan H X， Lei T， Chen K and Lu J. 2022. An explicit connection between independent vector analysis and tensor decomposition in blind source separation. IEEE Signal Processing Letters， 29： 1277-1281 ［DOI： 10.1109/LSP.2022.3176534http://dx.doi.org/10.1109/LSP.2022.3176534］
Sabour R M， Benezeth Y， De Oliveira P， Chappé J and Yang F. 2023. UBFC-Phys： a multimodal database for psychophysiological studies of social stress. IEEE Transactions on Affective Computing， 14（1）： 622-636 ［DOI： 10.1109/TAFFC.2021.3056960http://dx.doi.org/10.1109/TAFFC.2021.3056960］
Singh N and Raj A A B. 2020. Estimation of heart rate and respiratory rate using imaging photoplethysmography technique//Proceedings of 2020 International Conference on System， Computation， Automation and Networking （ICSCAN）. Pondicherry， India： IEEE： #9262320 ［DOI： 10.1109/ICSCAN49426.2020.9262320http://dx.doi.org/10.1109/ICSCAN49426.2020.9262320］
Song R C， Chen H， Cheng J， Li C， Liu Y and Chen X. 2021a. PulseGAN： learning to generate realistic pulse waveforms in remote photoplethysmography. IEEE Journal of Biomedical and Health Informatics， 25（5）： 1373-1384 ［DOI： 10.1109/JBHI.2021.3051176http://dx.doi.org/10.1109/JBHI.2021.3051176］
Song R C， Li J J， Cheng J， Li C， Liu Y and Chen X. 2021b. Motion robust imaging ballistocardiography through a two-step canonical correlation analysis. IEEE Transactions on Instrumentation and Measurement， 70： #4001710 ［DOI： 10.1109/TIM.2020.3034974http://dx.doi.org/10.1109/TIM.2020.3034974］
Song R C， Li J J， Wang M D， Cheng J， Li C and Chen X. 2021c. Remote photoplethysmography with an EEMD-MCCA method robust against spatially uneven illuminations. IEEE Sensors Journal， 21（12）： 13484-13494 ［DOI： 10.1109/JSEN.2021.3067770http://dx.doi.org/10.1109/JSEN.2021.3067770］
Špetlík R， Franc V， Čech J and Matas J. 2018. Visual heart rate estimation with convolutional neural network//Proceedings of the 29th British Machine Vision Conference. Newcastle， UK： ［s.n.］： 3-6
Tsou Y Y， Lee Y A， Hsu C T and Chang S H. 2020. Siamese-rPPG network： remote photoplethysmography signal estimation from face videos//The 35th Annual ACM Symposium on Applied Computing. Brno， Czech Republic： ACM： 2066-2073 ［DOI： 10.1145/3341105.3373905http://dx.doi.org/10.1145/3341105.3373905］
Wang W J， den Brinker A C and De Haan G. 2020. Discriminative signatures for remote-PPG. IEEE Transactions on Biomedical Engineering， 67（5）： 1462-1473 ［DOI： 10.1109/TBME.2019.2938564http://dx.doi.org/10.1109/TBME.2019.2938564］
Wang W J， Den Brinker A C， Stuijk S and Haan G D. 2017. Algorithmic principles of remote PPG. IEEE Transactions on Biomedical Engineering， 64（7）： 1479-1491 ［DOI： 10.1109/TBME.2016.2609282http://dx.doi.org/10.1109/TBME.2016.2609282］
Yang Z， Yang X Z， Huo L， Liu X N and Li J S. 2018. Heart rate estimation from face videos against motion interference. Journal of Electronics and Information Technology， 40（6）： 1345-1352
杨昭， 杨学志， 霍亮， 刘雪南， 李江山. 2018. 抗运动干扰的人脸视频心率估计. 电子与信息学报， 40（6）： 1345-1352 ［DOI： 10.11999/JEIT170824http://dx.doi.org/10.11999/JEIT170824］
Yu Z T， Li X B and Zhao G Y. 2021. Facial-video-based physiological signal measurement： recent advances and affective applications. IEEE Signal Processing Magazine， 38（6）： 50-58 ［DOI： 10.1109/MSP.2021.3106285http://dx.doi.org/10.1109/MSP.2021.3106285］
Zhao C C， Zhou M H， Han W R and Feng Y J. 2022. Anti-motion remote measurement of heart rate based on region proposal generation and multi-scale ROI fusion. IEEE Transactions on Instrumentation and Measurement， 71： #5012413 ［DOI： 10.1109/TIM.2022.3169567http://dx.doi.org/10.1109/TIM.2022.3169567］
相关文章
相关作者
相关机构
京公网安备11010802024621