浏览全部资源
扫码关注微信
Published： 16 January 2024 ，
移动端阅览
阳强， 罗坚， 黄宇琛. 2024. 遮挡条件下的步态图像时空修复网络及其应用. 中国图象图形学报， 29(01):0179-0191
Yang Qiang， Luo Jian， Huang Yuchen. 2024. Gait image spatio-temporal restoration network and its application under occlusion conditions. Journal of Image and Graphics， 29(01):0179-0191
目的
2
当前，基于视觉的步态识别方法多基于完整的步态序列图像。然而，现实场景拍摄下的行人难免被遮挡，以至于获取的步态图像不完整，对识别结果有很大影响。如何处理大面积遮挡是步态识别中一个具有挑战性且重要的问题。针对此，提出了一种步态时空序列重建网络（gait spatio-temporal reconstruction network，GSTRNet），用于修复被遮挡的步态序列图像。
方法
2
使用基于3D卷积神经网络和Transformer的GSTRNet来修复步态序列，在修复每一帧步态图像的空间信息的同时保持帧与帧之间的时空连贯性。GSTRNet通过引入YOLOv5（you only look once）网络来检测步态图像的局部遮挡区域，并将其作为先验知识为遮挡修复区域分配更高的修复权值，实现遮挡区域的局部修复，将局部修复步态图与原始遮挡图像进行融合，生成完整的修复步态图。同时，在GSTRNet中引入三元组特征损失和重建损失组成的联合损失函数来优化修复网络，提升修复效果。最终，以修复完整的步态序列图像为特征进行身份识别。
结果
2
本文在大规模步态数据集OU_MVLP（the OU-ISIR gait database，multi-view large population dataset）中人工合成遮挡步态序列数据来进行修复实验。结果表明，该方法在面对步态轮廓大面积遮挡时，识别准确率比现有的步态修复和遮挡识别方法有一定的提升，如在未知遮挡模式时比三元组视频生成对抗网络（sequence video wasserstein generative adversarial network based on triplet hinge loss，sVideoWGAN-hinge）最高提升6.7%，非单一模式遮挡时比Gaitset等方法识别率提高40%左右。
结论
2
本文提出的GSTRNet对各种遮挡模式下的步态图像序列有较好的修复效果，使用修复后图像进行步态识别，可有效改善识别率。
Objective
2
Gait recognition is a kind of identity recognition method based on human walking mode， which has been widely used in the field of video surveillance and public security. Compared with the face， fingerprint， and other biometric features， it has the advantages of long-distance recognition without the need for participant cooperation and the difficulty of camouflaging and hiding. At present， gait recognition algorithms are based on vision and deep learning， most of which use gait sequences without occlusion to form gait features for recognition. However， in reality， people under the monitoring of various public places are inevitably blocked， so the gait sequences obtained are usually occluded. The occlusion sequence has a great effect on gait recognition， such as the inability to obtain accurate gait periods from the sequence， and the lack of gait spatio-temporal information is also more serious， which leads to a substantial reduction in recognition performance. The existing occlusion gait processing algorithmsare divided into two kinds： One is to extract directly the features of occlusion robustness to identify from the occlusion sequence. It often needs to know the gait period in advance， but it is difficult in the occlusion gait sequence. The other algorithms perform identification by reconstructing the gait silhouette or repairing the gait features， but the existing algorithms often have poor performance when the occlusion area is large or the entire sequence is occluded.
Method
2
A gait spatio-temporal reconstruction network （GSTRNet）， which consists of the occlusion detection network you only look once （YOLO）， the spatio-temporal codec network， and the feature extraction network （Gaitset）， based on prior knowledge is proposed to repair occluded gait sequences. GSTRNet uses YOLOv5 to detect the occlusion region in sequence （assigning the occlusion area to 1 and the nonocclusion area to 0） as a piece of prior knowledge to assign a higher weight to the loss of the occlusion area. Spatiotemporal codec network consists of 3D convolutional neural network （3DCNN） and Transformers. The 3DCNN can repair the spatial information of each gait image while maintaining the time coherence between frames. The encoder uses 3DCNN with a stride of 2 to reduce the dimension of the data so that each element can participate in the current sampling and more detailed information can be retained. The decoder uses skip connection to stitch together features in the encoder to reduce further the loss of detail due to encoder down sampling. To ensure the time and space consistency of the entire repair sequence， multiple Transformers composed of multiscale self-attention module are added between the encoder and decoder， extracting useful information from the global and local scope to repair the gait sequence. Because the 3DCNN is a global repair， the nonocclusion region data in the repaired gait sequence also change， and GSTRNet uses prior knowledge to take only the occlusion region repair result from the decoder output and then adds it to the original sequence as the output of the network. The Gaitset network is also introduced to extract features from three sequences as triplet losses to maintain feature consistency between the repair sequence and the original sequence， namely， occlusion sequences， genuine sequences （other nonocclusion sequences with the same identity as occlusion sequences）， and imposter sequences （sequences that have different identities from occlusion sequences）. In the OU-ISIR gait database， multiview large population dataset （OU_MVLP）， 24 occlusion gait sequences are synthesized as experimental data by simulating various occlusion types in real life， and our algorithm is evaluate dusing three sets of experiments： the occlusion mode is known and the gallery and probe occlusion modes are consistent， the occlusion mode is known but the gallery and probe occlusion modes are inconsistent， and the occlusion mode is unknown.
Result
2
Results show the proposed algorithm performs better than the existing occlusion sequence repair algorithms. Compared with other repair algorithms， the rank1 recognition rates of our algorithm in single occlusion mode and non-single occlusion mode when the occlusion mode is known are 4.1% and 4.1% higher， respectively and have a maximum recognition accuracy improvement of 6.7% in the case of large area occlusion of the gait silhouette when the occlusion mode is unknown. Compared with gait recognition algorithms such as 3D local convolutional neural networks， the recognition rate in non-single occlusion mode has a maximum increase of about 50%.
Conclusion
2
The proposed GSTRNet model has a good effect on the repair of gait sequences to varying degrees in various occlusion modes and has a strong feasibility in reality.
步态识别步态轮廓修复先验知识三维卷积神经网络（3DCNN）Transformer
gait recognitionoccluded silhouette reconstructionprior knowledgethree-dimensional convolutional neural network（3DCNN）Transformer
Babaee M， Li L and Rigoll G. 2018. Gait recognition from incomplete gait cycle//Proceedings of the 25th IEEE International Conference on Image Processing （ICIP）. Athens， Greece： IEEE： 768-772 ［DOI： 10.1109/ICIP.2018.8451785http://dx.doi.org/10.1109/ICIP.2018.8451785］
Babaee M， Zhu Y， Kopuklu O， Hormann S and Rigoll G. 2019. Gait energy image restoration using generative adversarial networks//Proceedings of 2019 IEEE International Conference on Image Processing （ICIP）. Taipei， China， IEEE： 2596-2600 ［DOI： 10.1109/ICIP.2019.8803236http://dx.doi.org/10.1109/ICIP.2019.8803236］
Ben X Y， Xu S and Wang K J. 2012. Review on pedestrian gait feature expression and recognition. Pattern Recognition and Artificial Intelligence， 25（1）： 71-81
贲晛烨， 徐森， 王科俊. 2012. 行人步态的特征表达及识别综述. 模式识别与人工智能， 25（1）： 71-81 ［DOI： 10.16451/j.cnki.issn1003-6059.2012.01.010http://dx.doi.org/10.16451/j.cnki.issn1003-6059.2012.01.010］
Bochkovskiy A， Wang C Y and Liao H Y M. 2020. YOLOv4： optimal speed and accuracy of object detection ［EB/OL］ ［2022-12-03］. https://arxiv.org/pdf/2004.10934.pdfhttps://arxiv.org/pdf/2004.10934.pdf
Chao H Q， He Y W， Zhang J P and Feng J F. 2019. GaitSet： regarding gait as a set for cross-view gait recognition. Proceedings of the AAAI Conference on Artificial Intelligence， 33（1）： 8126-8133 ［DOI： 10.1609/aaai.v33i01.33018126http://dx.doi.org/10.1609/aaai.v33i01.33018126］
Chattopadhyay P， Sural S and Mukherjee J. 2015. Frontal gait recognition from occluded scenes. Pattern Recognition Letters， 63： 9-15 ［DOI： 10.1016/j.patrec.2015.06.004http://dx.doi.org/10.1016/j.patrec.2015.06.004］
Chikano M， Konno T and Awai S. 2021. Robust gait recognition for occlusion caused by surveillance cameras//Proceedings of 2021 IEEE 10th Global Conference on Consumer Electronics （GCCE）. Kyoto， Japan： IEEE： 148-149 ［DOI： 10.1109/GCCE53005.2021.9622020http://dx.doi.org/10.1109/GCCE53005.2021.9622020］
Choi S， Kim J， Kim W and Kim C. 2019. Skeleton-based gait recognition via robust frame-level matching. IEEE Transactions on Information Forensics and Security， 14（10）： 2577-2592 ［DOI： 10.1109/TIFS.2019.2901823http://dx.doi.org/10.1109/TIFS.2019.2901823］
Das D， Agarwal A， Chattopadhyay P and Wang L P. 2020. RGait-NET： an effective network for recovering missing information from occluded gait cycles ［EB/OL］. ［2022-12-03］. https://arxiv.org/pdf/1912.06765.pdfhttps://arxiv.org/pdf/1912.06765.pdf
Fan C， Peng Y J， Cao C S， Liu X， Hou S H， Chi J N， Huang Y Z， Li Q and He Z Z. 2020. GaitPart： temporal part-based model for gait recognition//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition （CVPR）. Seattle， USA： IEEE： 14213-14221 ［DOI： 10.1109/CVPR42600.2020.01423http://dx.doi.org/10.1109/CVPR42600.2020.01423］
Gupta A and Semwal V B. 2022. Occluded gait reconstruction in multi person gait environment using different numerical methods. Multimedia Tools and Applications， 81（16）： 23421-23448 ［DOI： 10.1007/s11042-022-12218-2http://dx.doi.org/10.1007/s11042-022-12218-2］
He K M， Zhang X Y， Ren S Q and Sun J. 2015. Delving deep into rectifiers： surpassing human-level performance on imageNet classification//Proceedings of 2015 IEEE International Conference on Computer Vision （ICCV）. Santiago， Chile： IEEE： 1026-1034 ［DOI： 10.1109/ICCV.2015.123http://dx.doi.org/10.1109/ICCV.2015.123］
Huang X H， Zhu D W， Wang H， Wang X G， Yang B， He B T， Liu W Y and Feng B. 2021a. Context-sensitive temporal feature learning for gait recognition//Proceedings of 2021 IEEE/CVF International Conference on Computer Vision （ICCV）. Montreal， Canada： IEEE： 12889-12898 ［DOI： 10.1109/ICCV48922.2021.01267http://dx.doi.org/10.1109/ICCV48922.2021.01267］
Huang Z， Xue D X， Shen X， Tian X M， Li H Q， Huang J Q and Hua X S. 2021b. 3D local convolutional neural networks for gait recognition//Proceedings of 2021 IEEE/CVF International Conference on Computer Vision （ICCV）. Montreal， Canada： IEEE： 14900-14909 ［DOI： 10.1109/ICCV48922.2021.01465http://dx.doi.org/10.1109/ICCV48922.2021.01465］
Iwama H， Okumura M， Makihara Y and Yagi Y. 2012. The OU-ISIR gait database comprising the large population dataset and performance evaluation of gait recognition. IEEE Transactions on Information Forensics and Security， 7（5）： 1511-1521 ［DOI： 10.1109/TIFS.2012.2204253http://dx.doi.org/10.1109/TIFS.2012.2204253］
Kratzwald B， Huang Z W， Paudel D P， Dinesh A and Van Gool L. 2018. Improving video generation for multi-functional applications ［EB/OL］. ［2022-12-03］. https://arxiv.org/pdf/1711.11453.pdfhttps://arxiv.org/pdf/1711.11453.pdf
Kumara S S， Chattopadhyaya P and Wang L P. 2021. BGaitR-Net： occluded gait sequence reconstruction with temporally constrained model for gait recognition ［EB/OL］. ［2022-12-03］. https://arxiv.org/pdf/2110.09564.pdfhttps://arxiv.org/pdf/2110.09564.pdf
Lee T K M， Belkhatir M and Sanei S. 2009. Coping with full occlusion in fronto-normal gait by using missing data theory//Proceedings of the 7th International Conference on Information， Communications and Signal Processing （ICICS）. Macau， China： IEEE： 1-5 ［DOI： 10.1109/ICICS.2009.5397506http://dx.doi.org/10.1109/ICICS.2009.5397506］
Lin B B， Zhang S L and Bao F. 2020. Gait recognition with multiple-temporal-scale 3D convolutional neural network//Proceedings of the 28th ACM International Conference on Multimedia. Seattle： USA： ACM： 3054-3062 ［DOI： 10.1145/3394171.3413861http://dx.doi.org/10.1145/3394171.3413861］
Nangtin P， Kumhom P and Chamnongthai K. 2016. Gait identification with partial occlusion using six modules and consideration of occluded module exclusion. Journal of Visual Communication and Image Representation， 36： 107-121 ［DOI： 10.1016/j.jvcir.2016.01.008http://dx.doi.org/10.1016/j.jvcir.2016.01.008］
Ortells J， Mollineda R A， Mederos B and Martín-Félez R. 2017. Gait recognition from corrupted silhouettes： a robust statistical approach. Machine Vision and Applications， 28（1）： 15-33 ［DOI： 10.1007/s00138-016-0798-yhttp://dx.doi.org/10.1007/s00138-016-0798-y］
Uddin M Z， Muramatsu D， Takemura N， Ahad M A R and Yagi Y. 2019. Spatio-temporal silhouette sequence reconstruction for gait recognition against occlusion. IPSJ Transactions on Computer Vision and Applications， 11（1）： #9 ［DOI： 10.1186/s41074-019-0061-3http://dx.doi.org/10.1186/s41074-019-0061-3］
Vondrick C， Pirsiavash H and Torralba A. 2016. Generating videos with scene dynamics ［EB/OL］. ［2022-12-03］. https://arxiv.org/pdf/1609.02612.pdfhttps://arxiv.org/pdf/1609.02612.pdf
Weissenborn D， Täckström O and Uszkoreit J. 2020. Scaling autoregressive video models ［EB/OL］. ［2022-12-03］. https://arxiv.org/pdf/1906.02634.pdfhttps://arxiv.org/pdf/1906.02634.pdf
Xu W Z， Huang T H， Ben X Y， Zeng Y and Zhang J P. 2023. Cross-view gait recognition： a review. Journal of Image and Graphics， 28（5）： 1265-1286
许文正， 黄天欢， 贲晛烨， 曾翌， 张军平. 2023. 跨视角步态识别综述. 中国图象图形学报， 28（5）： 1265-1286 ［DOI： 10.11834/jig.220458http://dx.doi.org/10.11834/jig.220458］
Zhang Z Y， Tran L， Yin X， Atoum Y， Liu X M， Wan J and Wang N X. 2019. Gait recognition via disentangled representation learning//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition （CVPR）. Long Beach， USA： IEEE： 4705-4714 ［DOI： 10.1109/CVPR.2019.00484http://dx.doi.org/10.1109/CVPR.2019.00484］
相关文章
相关作者
相关机构
京公网安备11010802024621