浏览全部资源
扫码关注微信
Published： 16 May 2024 ，
移动端阅览
李伟， 黄娅， 张馨渊， 韩贵金. 2024. 针对遮挡物体的轮廓细化实例分割. 中国图象图形学报， 29(05):1221-1232
Li Wei， Huang Ya， Zhang Xinyuan， Han Guijin. 2024. Contour refinement instance segmentation for occluded objects. Journal of Image and Graphics， 29(05):1221-1232
目的
2
遮挡物体实例分割效果的好坏与物体轮廓的预测结果息息相关，但目前算法预测的物体轮廓并不够细化，使得分割掩膜粗糙，物体边界分割效果不佳。为此，以BCNet（bilayer convolutional network）网络为基础，提出一种针对遮挡物体的轮廓细化实例分割算法，预测的物体轮廓更加精细，分割掩膜更加完整。
方法
2
1）提出一种均衡池化注意力模块来提取特征，在传统一维平均池化的基础上，增加一维最大池化操作以突出细节特征，并将最大池化和平均池化结果进行加权融合来提取特征，使提取的特征能更好地兼顾物体的整体和边缘细节；2）将BCNet掩膜头中轮廓预测与掩膜预测分成两个支路来进行，从特征金字塔最高分辨率特征中提取感兴趣区域（region of interest， RoI）特征用于轮廓预测，并提出一种自适应特征融合模块，将轮廓预测支路中的特征与掩膜预测支路的特征进行融合，在轮廓预测支路中，融合掩膜预测支路的特征可以更好地判定轮廓所属物体类别，在掩膜预测支路，融合轮廓预测支路的特征能够更好地辅助掩膜定位。
结果
2
在COCO 2017（common objects in context 2017）数据集上，本文相较于目前同类网络中性能最优的 BCNet 网络，在骨干网络为ResNet-50/101（deep residual network）时平均精度（average precision， AP）分别提高了1.7%和2.1%。结合可视化结果，本文分割算法对遮挡物体的轮廓分割更加精细，能有效分割出更加完整、精细的掩码。
结论
2
提出的针对遮挡物体的轮廓细化实例分割算法，明显提升了遮挡物体实例分割的效果。
Objective
2
Instance segmentation is a popular topic in image processing and computer vision. It is widely used in unmanned driving， medical image segmentation， and other fields. The high overlap between objects in practical application scenarios makes instance segmentation more challenging. Instance segmentation models include two-stage and single-stage models. In a two-stage instance segmentation model， the two-layer instance segmentation network （i.e.， bilayer convolutional network （BCNet） is one of the most successful networks in the field of occlusion object instance segmentation. In contrast with previous two-stage instance segmentation methods， BCNet extracts regions of interest （RoI） and simultaneously regresses the occluder and occludee regions， grouping pixels that belong to the occluder region and treating them as pixels in the occludee region， but dividing them into two independent image layers. Consequently， the boundaries of the two objects are naturally decoupled， considering the interaction between them during the mask regression phase. Thus， a two-layer convolutional network is proposed， in which the top graph convolutional network （GCN） layer detects occluder objects while the bottom GCN layer infers occludee instances. The explicit modeling of occlusion relationships on the basis of a two-layer structure naturally decouples the boundaries of occluder and occludee instances， and considers the interaction between them in mask regression. BCNet proposes a new structure for highly overlapping objects in segmentation scenes. The segmentation performance of occluded object instances is closely related to the prediction results of object contours. However， the current algorithm’s predicted object contours are not sufficiently refined， resulting in rough segmentation masks and poor object boundary segmentation performance. Therefore， this study presents a contour refinement instance segmentation method for occluded objects based on BCNet. The predicted object contour is more precise， and the segmentation mask is more complete.
Method
2
Simple average pooling causes the loss of a significant amount of feature information and cannot generate more detailed feature information， while maximum pooling can retain more significant feature information. On this basis， the study proposes a balance pooling attention module. Global context information is collected in the form of 1D maximum pooling and 1D average pooling in parallel. The proportion of maximum pooling and average pooling is balanced through weighted feature fusion to generate more detailed information without losing overall features. The balance pooling attention module is added to ResNet to enable the feature extraction stage to extract features with more details. Features at different scales contain different feature information. High-resolution features contain more detailed information， while low-resolution features contain richer semantic information. In contrast with the input for the mask head in BCNet， this study separates the contour prediction branch from the mask prediction branch and adopts different feature inputs， extracting region of interest （RoI） features from the highest resolution features in a feature pyramid network （FPN） as features for the contour prediction branch and RoI features from the cascaded low-resolution features as input for the mask prediction branch. The features of the contour prediction branch have more detailed and spatial information， and the mask prediction branch has more abundant semantic information. To improve the segmentation effect further， the adaptive feature fusion module is proposed to fuse the features in the contour prediction branch with the features in the mask prediction branch. In the contour prediction branch， the features of the fused mask prediction branch can better determine the object category to which the contour belongs. In the mask prediction branch， the features of the fused contour prediction branch can better assist in mask positioning. The criss-cross attention module is used to infer the contours and masks of the occluder and occludee objects. It can maintain prediction accuracy and effectively reduce computational complexity compared with the non-local operator structure GCN module adopted in BCNet.
Result
2
Segmentation accuracy is compared with classic instance segmentation models， such as mask region-based convolutional neural network（Mask R-CNN）， CenterMask， and BCNet， on the common objects in context（COCO） 2017 validation set. Compared with the baseline network BCNet， average precision（AP） increases by 1.7% and 2.1% when the backbone network is ResNet-50 and ResNet-101， respectively. An evident improvement is observed in the segmentation accuracy of multi-scale targets， proving that this model is effective for improving the baseline network. The segmentation accuracy in this study is higher than those of two-stage instance segmentation algorithms， such as Mask R-CNN and CenterMask. The proposed method also exhibits advantages over single-stage segmentation algorithms， such as you only look at coefficients（YOLACT）. From the contour binary graph， the boundaries extracted using this method are more precise. The experimental results show that the proposed method exhibits strong model generalization and robustness.
Conclusion
2
The occluded object instance segmentation based on balance pooling attention and adaptive feature fusion proposed in this study can effectively improve the effectiveness of occluded object instance segmentation.
遮挡物体实例分割均衡池化注意力模块（BPAM）自适应特征融合模块（AFFM）BCNet轮廓预测支路掩膜预测支路
occluded objects instance segmentationbalance pooling attention module（BPAM）adaptive feature fusion module （AFFM）bilayer convolutional network （BCNet）contour prediction branchmask prediction branch
Bolya D， Zhou C， Xiao F Y and Lee Y J. 2019. YOLACT： real-time instance segmentation//Proceedings of 2019 IEEE/CVF International Conference on Computer Vision. Seoul， Korea （South）： IEEE： 9156-9165 ［DOI： 10.1109/ICCV.2019.00925http://dx.doi.org/10.1109/ICCV.2019.00925］
Cheng T H， Wang X G， Huang L C and Liu W Y. 2020. Boundary-preserving mask R-CNN//Proceedings of the 16th European Conference on Computer Vision. Glasgow， UK： Springer： 660-676 ［DOI： 10.1007/978-3-030-58568-6_39http://dx.doi.org/10.1007/978-3-030-58568-6_39］
Ehsani K， Mottaghi R and Farhadi A. 2018. SeGAN： segmenting and generating the invisible//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City， USA： IEEE： 6144-6153 ［DOI： 10.1109/CVPR.2018.00643http://dx.doi.org/10.1109/CVPR.2018.00643］
Goodfellow I J， Pouget-Abadie J， Mirza M， Xu B， Warde-Farley D， Ozair S， Courville A and Bengio Y. 2014. Generative adversarial nets//Proceedings of the 27th International Conference on Neural Information Processing Systems. Montreal， Canada： MIT Press
He K M， Gkioxari G， Dollr P and Girshick R. 2017. Mask R-CNN//Proceedings of 2017 IEEE International Conference on Computer Vision. Venice， Italy： IEEE： 2980-2988 ［DOI： 10.1109/ICCV.2017.322http://dx.doi.org/10.1109/ICCV.2017.322］
He K M， Zhang X Y， Ren S Q and Sun J. 2016. Deep residual learning for image recognition//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas， USA： IEEE： 770-778 ［DOI： 10.1109/CVPR.2016.90http://dx.doi.org/10.1109/CVPR.2016.90］
Huang Z J， Huang L C， Gong Y C， Huang C and Wang X G. 2019a. Mask scoring R-CNN//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach， USA： IEEE： 6409-6418 ［DOI： 10.1109/CVPR.2019.00657http://dx.doi.org/10.1109/CVPR.2019.00657］
Huang Z L， Wang X G， Huang L C， Huang C， Wei Y C and Liu W Y. 2019b. CCNet： criss-cross attention for semantic segmentation//Proceedings of 2019 IEEE/CVF International Conference on Computer Vision. Seoul， Korea （South）： IEEE： 603-612 ［DOI： 10.1109/ICCV.2019.00069http://dx.doi.org/10.1109/ICCV.2019.00069］
Ke L， Tai Y W and Tang C K. 2021. Deep occlusion-aware instance segmentation with overlapping bilayers//Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville， USA： IEEE： 4019-4028 ［DOI： 10.1109/CVPR46437.2021.00401http://dx.doi.org/10.1109/CVPR46437.2021.00401］
Kipf T N and Welling M. 2017. Semi-supervised classification with graph convolutional networks//Proceedings of the 5th International Conference on Learning Representations. Toulon， France： OpenReview.net
Kirillov A， Wu Y X， He K M and Girshick R. 2020. PointRend： image segmentation as rendering//Proceeding of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle， USA： IEEE： 9796-9805 ［DOI： 10.1109/CVPR42600.2020.00982http://dx.doi.org/10.1109/CVPR42600.2020.00982］
Lazarow J， Lee K， Shi K Y and Tu Z W. 2020. Learning instance occlusion for panoptic segmentation//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle， USA： IEEE： 10717-10726 ［DOI： 10.1109/CVPR42600.2020.01073http://dx.doi.org/10.1109/CVPR42600.2020.01073］
Lee Y and Park J. 2020. CenterMask： real-time anchor-free instance segmentation//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle， USA： IEEE： 13906-13915 ［DOI： 10.1109/CVPR42600.2020.01392http://dx.doi.org/10.1109/CVPR42600.2020.01392］
Lin T Y， Dollr P， Girshick R， He K M， Hariharan B and Belongie S. 2017. Feature pyramid networks for object detection//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu， USA： IEEE： 936-944 ［DOI： 10.1109/CVPR.2017.106http://dx.doi.org/10.1109/CVPR.2017.106］
Lin T Y， Maire M， Belongie S， Hays J， Perona P， Ramanan D， Dollr P and Zitnick C L. 2014. Microsoft COCO： common objects in context//Proceedings of the 13th European Conference on Computer Vision. Zurich， Switzerland： Springer， 740-755 ［DOI： 10.1007/978-3-319-10602-1_48http://dx.doi.org/10.1007/978-3-319-10602-1_48］
Liu S， Qi L， Qin H F， Shi J P and Jia J Y. 2018. Path aggregation network for instance segmentation//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City， USA： IEEE： 8759-8768 ［DOI： 10.1109/CVPR.2018.00913http://dx.doi.org/10.1109/CVPR.2018.00913］
Milletari F， Navab N and Ahmadi S A. 2016. V-Net： fully convolutional neural networks for volumetric medical image segmentation//Proceedings of the 4th International Conference on 3D Vision. Stanford， USA： IEEE： 565-571 ［DOI： 10.1109/3DV.2016.79http://dx.doi.org/10.1109/3DV.2016.79］
Ren S Q， He K M， Girshick R and Sun J. 2017. Faster R-CNN： towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence， 39（6）： 1137-1149 ［DOI： 10.1109/TPAMI.2016.2577031http://dx.doi.org/10.1109/TPAMI.2016.2577031］
Tian Z， Shen C H and Chen H. 2020. Conditional convolutions for instance segmentation//Proceedings of the 16th European Conference on Computer Vision. Glasgow， UK： Springer： 282-298 ［DOI： 10.1007/978-3-030-58452-8_17http://dx.doi.org/10.1007/978-3-030-58452-8_17］
Tian Z， Shen C H， Chen H and He T. 2019. FCOS： fully convolutional one-stage object detection//Proceedings of 2019 IEEE/CVF International Conference on Computer Vision. Seoul， Korea （South）： IEEE： 9626-9635 ［DOI： 10.1109/ICCV.2019.00972http://dx.doi.org/10.1109/ICCV.2019.00972］
Wan X J， Zhou Y Y， Shen M F， Zhou T and Hu F Y. 2023. Multi-scale context information fusion for instance segmentation. Journal of Image and Graphics， 28（2）： 495-509
万新军， 周逸云， 沈鸣飞， 周涛， 胡伏原. 2023. 融合多尺度上下文信息的实例分割. 中国图象图形学报， 28（2）： 495-509 ［DOI： 10.11834/jig.211090http://dx.doi.org/10.11834/jig.211090］
Wang X L， Girshick R， Gupta A and He K M. 2018. Non-local neural networks//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City， USA： IEEE ［DOI： 10.1109/CVPR.2018.00813http://dx.doi.org/10.1109/CVPR.2018.00813］
Wang X L， Kong T， Shen C H， Jiang Y N and Li L. 2020. SOLO： segmenting objects by locations//Proceedings of the 16th European Conference on Computer Vision. Springer： 649-665 ［DOI： 10.1007/978-3-030-58523-5_38http://dx.doi.org/10.1007/978-3-030-58523-5_38］
Yan X S， Yu Y L， Wang F G， Liu W X， He S F and Pan J. 2019. Visualizing the invisible： occluded vehicle segmentation and recovery//Proceedings of 2019 IEEE/CVF International Conference on Computer Vision. Seoul， Korea （South）： IEEE： 7617-7626 ［DOI： 10.1109/ICCV.2019.00771http://dx.doi.org/10.1109/ICCV.2019.00771］
Yuan X D， Kortylewski A， Sun Y H and Yuille A. 2021. Robust instance segmentation through reasoning about multi-object occlusion//Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville， USA： IEEE： 11136-11145 ［DOI： 10.1109/CVPR46437.2021.01099http://dx.doi.org/10.1109/CVPR46437.2021.01099］
Yuan Y H， Xie J Y， Chen X L and Wang J D. 2020. SegFix： model-agnostic boundary refinement for segmentation//Proceedings of the 16th European Conference on Computer Vision. Glasgow， UK： Springer： 489-506 ［DOI： 10.1007/978-3-030-58610-2_29http://dx.doi.org/10.1007/978-3-030-58610-2_29］
Zhan X H， Pan X G， Dai B， Liu Z W， Lin D H and Loy C C. 2020. Self-supervised scene de-occlusion//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle， USA： IEEE： 3784-3792 ［DOI： 10.1109/CVPR42600.2020.00384http://dx.doi.org/10.1109/CVPR42600.2020.00384］
Zhang G， Lu X， Tan J R， Li J M， Zhang Z X， Li Q Q and Hu X L. 2021. RefineMask： towards high-quality instance segmentation with fine-grained features//Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville， USA： IEEE： 6857-6865 ［DOI： 10.1109/CVPR46437.2021.00679http://dx.doi.org/10.1109/CVPR46437.2021.00679］
Zhou Y Y， Wan X J， Hu F Y and Chen H. 2023. Instance segmentation algorithm based on joint attention and feature association. Computer Engineering， 49（6）： 217-226
周逸云， 万新军， 胡伏原， 陈昊. 2023. 基于联合注意与特征关联的实例分割算法. 计算机工程， 49（6）： 217-226 ［DOI： 10.19678/j.issn.1000-3428.0064695http://dx.doi.org/10.19678/j.issn.1000-3428.0064695］
相关文章
相关作者
相关机构
京公网安备11010802024621