Precise positioning method of deformed workpiece by fusing edge and grayscale features

Li Sicong; Zhu Feng; Wu Qingxiao

doi:10.11834/jig.221183

Image Analysis and Recognition | Views : 0 下载量: 157 CSCD: 0

PDF
Export
Share
Collection
Album

Precise positioning method of deformed workpiece by fusing edge and grayscale features
Vol. 29, Issue 1, Pages: 192-204(2024)
Received：28 December 2022，

Revised：18 April 2023，

Published：16 January 2024
DOI： 10.11834/jig.221183
稿件说明：

移动端阅览

李思聪，朱枫，吴清潇. 2024. 融合边缘与灰度特征的形变工件精准定位方法. 中国图象图形学报， 29(01):0192-0204 DOI： 10.11834/jig.221183.

Li Sicong， Zhu Feng， Wu Qingxiao. 2024. Precise positioning method of deformed workpiece by fusing edge and grayscale features. Journal of Image and Graphics， 29(01):0192-0204 DOI： 10.11834/jig.221183.

摘要

目的

工业机器人视觉领域经常需要对一些由拼装、冲压或贴合等工艺造成的形变工件进行精准定位，工件的大部分特征表现出一定程度的非刚性，其他具备良好一致性的部分通常特征简单，导致一些常用的目标检测算法精度不足或鲁棒性不强，难以满足实际需求。针对这一问题，提出融合边缘与灰度特征的形变工件精准定位方法。

方法

第1阶段提出多归一化互相关的模板匹配MNCC（multi normalized cross correlation）方法检测形变目标，利用余弦距离下的灰度聚类获得均值模板，通过滑动窗口的方式，结合金字塔跟踪，自顶向下地优先搜索类均值模板，得到类匹配候选，然后进行类内细搜索获得最佳位置匹配。第2阶段提出一种改进的形状匹配方法T-SBM（truncated shape-based matching），通过改变原始SBM（shape-based matching）的梯度方向内积的计算方式，对负梯度极性方向截断，削弱目标背景不稳定导致局部梯度方向反转时对整体评分的负贡献，改善边缘稀疏或特征简单导致检测鲁棒性低的问题。最后提出二维高斯条件密度评价，将灰度特征、形状特征和形变量进行综合加权，获得理想目标评价，实现序贯检测。

结果

实验部分分别与SBM、归一化互相关匹配算法（normalized cross correlation，NCC）、LINE2D（linearizing the memory 2D）算法和YOLOv5s（you only look once version 5 small）算法在5种类型工件的472幅真实工业图像上进行了对比测试，在检出分值大于0.8（实际常用的阈值区间）时，提出算法的召回率优于其他几种测试算法；在IoU（intersection over union）阈值0.9时的平均检测准确率为81.7%，F1-Score为95%，两组指标相比其他测试算法分别至少提升了10.8%和8.3%。在平均定位精度方面，提出算法的定位偏差在IoU阈值0.9时达到了2.44像素，在5种测试算法中的表现也为最佳。

结论

提出了一种两阶段的定位方法，该方法适用于检测工业场景中由拼装、冲压和贴合等工艺制成的形变工件并能够进行精准定位，尤其适用于工业机器人视觉引导定位应用场景，并在实际项目中得到了应用。

Abstract

Objective

In industrial robot vision， accurately detecting deformed workpieces caused by assembly， stamping， or lamination is often necessary. These workpieces sometimes show non-rigid characteristics， such as dislocation or twist deformation. Most features do not maintain good shape consistency， while the remaining undeformed parts are generally simple， for example， sparse edges， which are not globally unique. In addition， obtaining massive training samples before workpieces are mass produced is not realistic. Hence， some commonly used object detection methods have insufficient accuracy or weak robustness， challenging meeting the actual needs.

Method

To address the problem， a two-stage precise positioning method of deformed workpieces by fusing edge and grayscale features is proposed. The first stage is the coarse position detection of deformed targets based on grayscale features， and the second stage is precise positioning based on shape features. The innovation of the first stage lies in that a multi normalized cross correlation （MNCC） matching method is proposed， which includes offline and online parts. In the offline part， the grayscale clustering algorithm at cosine distance is used to obtain the class-mean template， which characterizes a class center of similar features in the target deformation space. Therefore， fewer class-mean templates can be used to represent the grayscale features of the target’s deformation after discretization. In the online part， by sliding window combined with pyramid tracking， the class-mean template is searched preferentially from top to bottom to acquire the class-mean candidates. Then， a detailed search of the candidates within the class is carried out to obtain the best match， achieve the efficient matching of deformed workpieces， and complete the task of coarse position detection during the first stage. A truncated shape-based matching （T-SBM） method is proposed in the second stage to achieve precise positioning using the target edge. By changing the similarity measurement based on the gradient’s inner product， the gradient vector of opposite direction is truncated， so no negative evaluation of the local edge points exists. The improvement restricts the negative contribution on the overall similarity score when the local gradient direction is inverted due to the inconsistency of the target background. The simple representations of sparse edges leading to low robustness are prevented. Finally， a 2D Gaussian conditional density evaluation is proposed to combine grayscale features， shape features， and deformation quantity reasonably. The candidate with the top score wins the best match. The proposed evaluation provides a comprehensive estimation for the ideal target position detection under the 3-sigma principle， realizes the precise positioning of the deformed workpiece， and completes the sequential detection.

Result

In the experimental part， the proposed method is compared with classical shape-based matching （SBM）， normalized cross-correlation （NCC）， linearizing the memory 2D （LINE2D）， and you only look once version 5 small （YOLOv5s） on 472 authentic industrial images consisting of five types of workpieces， namely， TV back， led panel， screw hole， metal tray， and aluminum plate. Industry vision software， HALCON， provides the implementation of SBM and NCC， and LINE2D is from OpenCV. The evaluation contains F1-score， recall， detection accuracy， and average pixel distance， where the first three and the last regards detection robustness and positioning accuracy， respectively. At intersection over union （IoU） of 0.9， a strict enough threshold for precise positioning， the average detection accuracy and the F1-score of the proposed method are 81.7% and 95%， respectively， and improve by 10.8% and 8.3%， compared with other test methods. When the minscore threshold is less than 0.8， the recall of the proposed method is slightly inferior to that of the NCC method. However， when the minscore is greater than 0.8， a commonly used threshold interval， the proposed method substantially outperforms the other methods. In terms of average positioning accuracy， the positioning error based on the Euclidean distance of the proposed method is as low as 2.44 pixels at the IoU threshold of 0.9， which is muchbetter than the that of the other test methods.

Conclusion

A two-stage precision positioning method for deformed workpieces made by assembly， stamping， or lamination is proposed. In the experiment， the proposed method outperforms the other test methods on detection robustness and positioning accuracy， which shows the proposed method is suitable for precisely positioning deformed workpieces in industrial scenes.

关键词

Keywords

references

Bochkovskiy A ， Wang C Y and Liao H Y M . 2020 . YOLOv4： optimal speed and accuracy of object detection . ［EB/OL］. ［ 2022-12-28 ］. https://arxiv.org/pdf/2004.10934.pdf https://arxiv.org/pdf/2004.10934.pdf

Dong H X ， Prasad D K and Chen I M . 2021 . Object pose estimation via pruned Hough forest with combined split schemes for robotic grasp . IEEE Transactions on Automation Science and Engineering ， 18 （ 4 ）： 1814 - 1821 ［ DOI： 10.1109/TASE.2020.3021119 http://dx.doi.org/10.1109/TASE.2020.3021119 ］

Gioi R ， Jakubowicz J ， Morel J and Randall G . 2012 . LSD： a line segment detector. Image Process . On Line ， 02 ： 35 - 55 ［ DOI： 10.5201/IPOL.2012.gjmr-lsd http://dx.doi.org/10.5201/IPOL.2012.gjmr-lsd ］

Han B ， Mu Z F ， Le X F ， Jia X Z ， Shi X W and Li B B . 2018 . Fast recurrence algorithm for computing sub-image energy using normalized cross correlation . Optics and Precision Engineering ， 26 （ 10 ）： 2565 - 2574

韩冰，牟忠锋，乐小峰，贾小志，石选卫，李贝贝 . 2018 . 归一化互相关中计算基准子图能量的快速递推 . 光学精密工程， 26 （ 10 ）： 2565 - 2574 ［ DOI： 10.3788/OPE.20182610.2565 http://dx.doi.org/10.3788/OPE.20182610.2565 ］

He Z X ， Jiang Z W ， Zhao X Y ， Zhang S Y and Wu C R . 2020 . Sparse template-based 6-D pose estimation of metal parts using a monocular camera . IEEE Transactions on Industrial Electronics ， 67 （ 1 ）： 390 - 401 ［ DOI： 10.1109/TIE.2019.2897539 http://dx.doi.org/10.1109/TIE.2019.2897539 ］

Hinterstoisser S ， Cagniart C ， Ilic S ， Sturm P ， Navab N ， Fua P and Lepetit V . 2012 . Gradient response maps for real-time detection of textureless objects . IEEE Transactions on Pattern Analysis and Machine Intelligence ， 34 （ 5 ）： 876 - 888 ［ DOI： 10.1109/TPAMI.2011.206 http://dx.doi.org/10.1109/TPAMI.2011.206 ］

Lepetit V ， Moreno-Noguer F and Fua P . 2009 . EP n P： an accurate o（ n ） solution to the P n P problem . International Journal of Computer Vision ， 81 （ 2 ）： 155 - 166 ［ DOI： 10.1007/s11263-008-0152-6 http://dx.doi.org/10.1007/s11263-008-0152-6 ］

Lin T Y ， Goyal P ， Girshick R ， He K M and Dollar P . 2020 . Focal loss for dense object detection . IEEE Transactions on Pattern Analysis and Machine Intelligence ， 42 （ 2 ）： 318 - 327 ［ DOI： 10.1109/TPAMI.2018.2858826 http://dx.doi.org/10.1109/TPAMI.2018.2858826 ］

Liu L . 2019 . Research and Application of Mark Point Sub-Pixel Location Algorithms on Mobile Screen . Xiamen ： Xiamen University

刘磊 . 2019 . 手机屏幕Mark点亚像素定位算法研究及应用 . 厦门：厦门大学

Liu Z Q ， Wan P ， Ling L ， Chen L ， Li X F and Zhou W X . 2018 . Recognition and grabbing system for workpieces exceeding the visual field based on machine vision . Robot ， 40 （ 3 ）： 294 - 300 ， 308

刘正琼，万鹏，凌琳，陈莉，李学飞，周文霞 . 2018 . 基于机器视觉的超视场工件识别抓取系统 . 机器人， 40 （ 3 ）： 294- 300 ， 308 ［ DOI： 10.13973/j.cnki.robot.170365 http://dx.doi.org/10.13973/j.cnki.robot.170365 ］

Lu P X ， Yang K M ， Lu S and Zhu Y . 2021 . A high-precision positioning algorithm of alignment mark for wafer bonding . Chinese Journal of Scientific Instrument ， 42 （ 11 ）： 220 - 229

鲁沛昕，杨开明，鲁森，朱煜 . 2021 . 用于晶圆键合的对准标记定位算法 . 仪器仪表学报， 42 （ 11 ）： 220 - 229 ［ DOI： 10.19650/j.cnki.cjsi.J2108113 http://dx.doi.org/10.19650/j.cnki.cjsi.J2108113 ］

MVTec Software GmbH . 2020 . HALCON ［EB/OL］. ［ 2022-12-28 ］. https://www.mvtec.com/products/halcon/ https://www.mvtec.com/products/halcon/

Sadeghi M A and Forsyth D . 2014 . 30 Hz object detection with DPM V5//Proceedings of the 13th European Conference on Computer Vision . Zurich， Switzerland ： Springer ： 65 - 79 ［ DOI： 10.1007/978-3-319-10590-1_5 http://dx.doi.org/10.1007/978-3-319-10590-1_5 ］

Steger C . 2002 . Occlusion， clutter， and illumination invariant object recognition . International Archives of Photogrammetry and Remote Sensing， and Spatial Information Sciences ， 34 （ 3 A）： 345 - 350

Tombari F ， Franchi A and Di L . 2013 . BOLD features to detect texture-less objects // Proceedings of 2013 IEEE International Conference on Computer Vision . Sydney， Australia ： IEEE： 1265 - 1272 ［ DOI： 10.1109/ICCV.2013.160 http://dx.doi.org/10.1109/ICCV.2013.160 ］

Ulrich M ， Follmann P and Neudeck J H . 2019 . A comparison of shape-based matching with deep-learning-based object detection . Technisches Messen ， 86 （ 11 ）： 685 - 698 ［ DOI： 10.1515/teme-2019-0076 http://dx.doi.org/10.1515/teme-2019-0076 ］

Wang H M . 2019 . Research on Correlation Algorithms for Real-Time Measurement of Wavefront Slop of Solar Multilayer Conjugate Adaptive Optics . Chengdu ： Institute of Optics and Electronics， Chinese Academy of Sciences （王黄铭 . 2019. 太阳多层共轭自适应光学波前斜率实时测量相关算法研究 . 成都：中国科学院光电技术研究所）

Yan M ， Tao D P and Pu Y Y . 2022 . Texture-less object detection method for industrial components picking system . Journal of Image and Graphics ， 27 （ 8 ）： 2418 - 2429

闫明，陶大鹏，普园媛 . 2022 . 面向工业零件分拣系统的低纹理目标检测 . 中国图象图形学报， 27 （ 8 ）： 2418 - 2429 ［ DOI： 10.11834/jig.210088 http://dx.doi.org/10.11834/jig.210088 ］

Zhang W Y . 2020 . Sorting Robot Based on Vision Guidance . Dalian ： Dalian Jiaotong University

张韦昱 . 2020 . 基于视觉引导的分拣机器人 . 大连：大连交通大学［ DOI： 10.26990/d.cnki.gsltc.2020.000434 http://dx.doi.org/10.26990/d.cnki.gsltc.2020.000434 ］

Zhou W . 2020 . Optic disk detection approach based on adaptive multi-scale template matching . Information and Control ， 49 （ 2 ）： 154 - 162

周唯 . 2020 . 基于自适应多尺度模板匹配的视盘检测方法 . 信息与控制， 49 （ 2 ）： 154 - 162 ［ DOI： 10.13976/j.cnki.xk.2020.9334 http://dx.doi.org/10.13976/j.cnki.xk.2020.9334 ］

Zhu Z K ， Lyu S C ， Wang X and Zhao Q . 2021 . TPH-YOLOv5： improved YOLOv5 based on transformer prediction head for object detection on drone-captured scenarios // Proceedings of 2021 IEEE/CVF International Conference on Computer Vision Workshops . Montreal， Canada ： IEEE： 2778 - 2788 ［ DOI： 10.1109/ICCVW54120.2021.00312 http://dx.doi.org/10.1109/ICCVW54120.2021.00312 ］

Alert me when the article has been cited

提交

Survey of semiconductor wafer defect detection method based on machine vision

Vision-based LCD/OLED defect detection methods： a critical summary

The growth of UAV aerial images-related power lines detection： a literature review of 2023

The cross-view gait recognition analysis based on generative adversarial networks derived of self-attention mechanism

Target detection algorithm of aerial remote sensing based on feature enhancement technology