浏览全部资源
扫码关注微信
Published： 16 January 2024 ，
移动端阅览
房磊， 史泽林， 刘云鹏， 李晨曦， 赵恩波， 张英迪. 2024. 几何联合分段亮度的线阵图像配准. 中国图象图形学报， 29(01):0080-0094
Fang Lei， Shi Zelin， Liu Yunpeng， Li Chenxi， Zhao Enbo， Zhang Yingdi. 2024. Joint geometric and piecewise photometric line-scan image registration. Journal of Image and Graphics， 29(01):0080-0094
目的
2
以非平行于目标的姿态成像时，线阵相机采集的图像的几何变换规律与面阵相机不同，这导致面阵图像的几何变换模型及其直接配准方法无法实现线阵图像的配准；同时，亮度恒常假设无法解决大视场镜头引起的图像亮度衰减问题。因此，提出了一种几何联合分段亮度的线阵图像直接配准方法。
方法
2
根据线阵图像的几何变换模型和分段增益—偏置亮度模型，将线阵图像的配准问题表示为一个非线性最小二乘问题。采用高斯—牛顿法对配准问题中的几何变换参数和亮度变换参数联合进行优化；此外，针对以单位变换为初始值时配准图像存在较大几何误差致使优化不收敛，设计了一种初始值快速搜索策略。
结果
2
实验数据包含本文采集的线阵图像数据集和真实列车线阵图像。配准结果表明，采用本文方法配准后的标注点坐标均方根误差均小于1个像素，优于采用面阵图像几何变换模型的直接配准方法。算法对亮度变化具有更强的鲁棒性，提高了线阵图像配准的成功率。
结论
2
本文提出的几何联合分段亮度线阵图像配准方法可以精确、鲁棒地对齐非平行姿态线阵相机所采集的图像。
Objective
2
Image registration is a fundamental problem in computer vision and image processing. It aims to eliminate the geometric difference of an object in an image collected by different cameras at various times and poses. Image registration has been widely used in several visual applications， such as image tracking， image fusion， image analysis， and anomaly detection. Image registration methods can be classified into feature-based and direct registration methods. The former calculates the parameters in a geometric transformation model by extracting and matching features， such as corners or edges， while the latter directly uses image intensity to infer the parameters. Evidently， choosing a reasonable geometric transformation model is the key to image alignment. The principles of line-scan and area-scan cameras are identical， and both cameras conform to the principle of pin-hole imaging. However， the imaging model of a line-scan camera is different from that of an area-scan camera due to the characteristics of its sensor. With the same change in camera pose， the locations of the same 3D world points mapped to the two types of images are different. That is， the geometric transformation law of an object in the images caused by the pose change of the two types of cameras is different. When the image plane of a line-scan camera is nonparallel to the object plane， geometric transformation models commonly used for area-scan image registration， such as the rigid， affine， and projection transformation models， cannot conform to the geometric transformation law of line-scan images. The direct registration method based on the geometric transformation model of an area-scan image cannot realize the geometric alignment of a line-scan image. Moreover， most existing direct image registration methods for solving the image alignment problem is based on the brightness constancy assumption and only geometric transformation is considered. In real-world applications， the variation of brightness is unavoidable and the brightness constancy assumption cannot address the problem of brightness attenuation when capturing images with a large-angle lens. Therefore， the line-scan image registration problem， which estimates geometric and photometric transformations between two images， is considered. Moreover， a direct registration method for line-scan images based on geometric and piecewise photometric transformations is proposed in this study.
Method
2
First， the optimization objective function of line-scan image registration is constructed by using the sum of squares difference of image intensity. In accordance with the geometric transformation model of line-scan images and the piecewise gain-bias photometric transformation model， the registration problem of a line-scan image is expressed as a nonlinear least squares problem. Second， the Gauss-Newton method is used to optimize the geometric and photometric transformation parameters in the registration problem. The nonlinear optimization objective function is linearized by performing a first-order Taylor expansion. The Jacobian of the warp and photometric transformation is derived on the basis of the geometric transformation model of a line-scan image and the gain-bias model. Finally， to obtain the optimal geometric and photometric transformation parameters， the increments of the warp and photometric transformation are repeatedly computed until they are below the threshold in accordance with the normal equation. As the initial value， the identity warp cannot be guaranteed near the optimal solution， and the iteration does not converge during registration. This problem is solved by designing an initial value fast matching method that provides an initial solution closer to the optimal one. The process of the initial value fast matching method is as follows： fixed-size areas are selected from the four corners of the template image and then matched to the target image in the corresponding position. The minimum and maximum coordinates of the optimal matching position in the horizontal and vertical directions are selected. Then， the scale and translation factors in the horizontal and vertical directions are solved， and the result is regarded as the initial value for the iteration. The initial value provided by the initial value fast matching method reduces geometric difference between the template and target images， and the success rate of the registration method is improved.
Result
2
To verify the proposed line-scan image registration method， a line-scan image acquisition system was built to obtain line-scan images of a planar object under different imaging poses and illumination variations. The experimental data also included electric multiple units （EMU） train line-scan images， which were collected by a line-scan camera in a natural environment. The images collected by the line-scan image acquisition system and the EMU train line-scan images were annotated separately， and the root-mean-square error （RMSE） of the annotated point coordinates was used as the evaluation index of the geometric error. The performance of the initial value fast matching method was verified on the line-scan image dataset collected in this study. The geometric error between the template image and the warped target image based on the initial value provided by the fast template block matching method was smaller than that based on the identity warp. This finding indicates that the initial value provided by the initial value fast matching method is closer to the optimal solution of the geometric transformation. Through the registration experiments on the collected dataset and the EMU train line-scan image， the results show that the RMSE of the annotated point coordinates is less than 1 pixel， and registration accuracy is excellent.
Conclusion
2
Our algorithm is more robust to lighting changes， and it improves the success rate of line-scan image registration. The joint geometric and piecewise photometric line-scan image registration method proposed in this study can accurately align the images collected in practical application scenes. This condition is also a foundation for train anomaly detection based on line-scan images. Therefore， the direct registration method proposed in this study can accurately and robustly align line-scan images collected under nonparallel poses.
线阵相机线阵图像直接配准方法几何变换亮度变换
line-scan cameraline-scan imagedirect registration methodgeometric transformationphotometric transformation
Alismail H， Kaess M， Browning B and Lucey S. 2017. Direct visual odometry in low light using binary descriptors. IEEE Robotics and Automation Letters， 2（2）： 444-451 ［DOI： 10.1109/LRA.2016.2635686http://dx.doi.org/10.1109/LRA.2016.2635686］
Baker S and Matthews I. 2004. Lucas-Kanade 20 years on： a unifying framework. International Journal of Computer Vision， 56（3）： 221-255 ［DOI： 10.1023/B：VISI.0000011205.11775.fdhttp://dx.doi.org/10.1023/B：VISI.0000011205.11775.fd］
Cao S Y， Shen H L， Chen S J and Li C G. 2020. Boosting structure consistency for multispectral and multimodal image registration. IEEE Transactions on Image Processing， 29： 5147-5162 ［DOI： 10.1109/TIP.2020.2980972http://dx.doi.org/10.1109/TIP.2020.2980972］
Chen L， Ling H B， Shen Y， Zhou F， Wang P， Tian X and Chen Y W. 2019. Robust visual tracking for planar objects using gradient orientation pyramid. Journal of Electronic Imaging， 28（1）： #013007 ［DOI： 10.1117/1.JEI.28.1.013007http://dx.doi.org/10.1117/1.JEI.28.1.013007］
Chen S J， Shen H L， Li C G and Xin J H. 2018. Normalized total gradient： a new measure for multispectral image registration. IEEE Transactions on Image Processing， 27（3）： 1297-1310 ［DOI： 10.1109/TIP.2017.2776753http://dx.doi.org/10.1109/TIP.2017.2776753］
Chen Y， Zhang Q， Li W J， Shi Y J and Chen L. 2021. Consistent registration of remote sensing images in parametric synthesized spatial transformation network. Journal of Image and Graphics， 26（12）： 2964-2980
陈颖， 张祺， 李文举， 石艳娇， 陈磊. 2021. 参数合成空间变换网络的遥感图像一致性配准. 中国图象图形学报， 26（12）： 2964-2980 ［DOI： 10.11834/jig.200587http://dx.doi.org/10.11834/jig.200587］
Erives H and Fitzgerald G J. 2005. Automated registration of hyperspectral images for precision agriculture. Computers and Electronics in Agriculture， 47（2）： 103-119 ［DOI： 10.1016/j.compag.2004.11.016http://dx.doi.org/10.1016/j.compag.2004.11.016］
Fang L， Shi Z L， Li C X， Liu Y P and Zhao E B. 2022a. Geometric transformation modeling for line-scan images under different camera poses. Optical Engineering， 61（10）： #103103 ［DOI： 10.1117/1.OE.61.10.103103http://dx.doi.org/10.1117/1.OE.61.10.103103］
Fang L， Shi Z L， Liu Y P， Li C X and Zhao E B. 2022b. The geometric transformation model of two views based on the line-scan camera imaging model//Proceedings of the 15th International Conference on Intelligent Robotics and Applications. Harbin， China： Springer： 113-124 ［DOI： 10.1007/978-3-031-13841-6_11http://dx.doi.org/10.1007/978-3-031-13841-6_11］
Han B X， Lu H M， Yu Q H and Zhang L L. 2021. Vanishing point estimation based on non-linear optimization in Manhattan world environments. Journal of Image and Graphics， 26（12）： 2931-2940
韩冰心， 卢惠民， 于清华， 张礼廉. 2021. 曼哈顿世界环境下消失点非线性优化估计算法. 中国图象图形学报， 26（12）： 2931-2940 ［DOI： 10.11834/jig.200398http://dx.doi.org/10.11834/jig.200398］
Iwasaki A and Fujisada H. 2005. ASTER geometric performance. IEEE Transactions on Geoscience and Remote Sensing， 43（12）： 2700-2706 ［DOI： 10.1109/TGRS.2005.849055http://dx.doi.org/10.1109/TGRS.2005.849055］
Jia D， Zhu N D， Yang N H， Wu S， Li Y X and Zhao M Y. 2019. Image matching methods. Journal of Image and Graphics， 24（5）： 677-699
贾迪， 朱宁丹， 杨宁华， 吴思， 李玉秀， 赵明远. 2019. 图像匹配方法研究综述. 中国图象图形学报， 24（5）： 677-699 ［DOI： 10.11834/jig.180501http://dx.doi.org/10.11834/jig.180501］
Laliberte A S， Goforth M A， Steele C M and Rango A. 2011. Multispectral remote sensing from unmanned aircraft： image processing workflows and applications for rangeland environments. Remote Sensing， 3（11）： 2529-2551 ［DOI： 10.3390/rs3112529http://dx.doi.org/10.3390/rs3112529］
Li C X， Shi Z L and Liu Y P. 2016. Joint geometric and photometric direct image registration based on Lie algebra parameterization//Proceedings Volume 10157， Infrared Technology and Applications， and Robot Sensing and Advanced Control. Beijing， China： SPIE： 465-471 ［DOI： 10.1117/12.2246720http://dx.doi.org/10.1117/12.2246720］
Li C X， Shi Z L， Liu Y P， Liu T C and Xu L Y. 2018. Efficient and robust direct image registration based on joint geometric and photometric Lie algebra. IEEE Transactions on Image Processing， 27（12）： 6010-6024 ［DOI： 10.1109/TIP.2018.2864895http://dx.doi.org/10.1109/TIP.2018.2864895］
Li J Y， Hu Q W and Ai M Y. 2020. RIFT： multi-modal image matching based on radiation-variation insensitive feature transform. IEEE Transactions on Image Processing， 29： 3296-3310 ［DOI： 10.1109/TIP.2019.2959244http://dx.doi.org/10.1109/TIP.2019.2959244］
Li X F， Wang L L， Wang J and Zhang X L. 2017. Multi-focus image fusion algorithm based on multilevel morphological component analysis and support vector machine. IET Image Processing， 11（10）： 919-926 ［DOI： 10.1049/iet-ipr.2016.0661http://dx.doi.org/10.1049/iet-ipr.2016.0661］
Liu C C. 2006. Processing of FORMOSAT-2 daily revisit imagery for site surveillance. IEEE Transactions on Geoscience and Remote Sensing， 44（11）： 3206-3214 ［DOI： 10.1109/TGRS.2006.880625http://dx.doi.org/10.1109/TGRS.2006.880625］
Liu L， Zhou F Q and He Y Z. 2016. Automated visual inspection system for bogie block key under complex freight train environment. IEEE Transactions on Instrumentation and Measurement， 65（1）： 2-14 ［DOI： 10.1109/TIM.2015.2479101http://dx.doi.org/10.1109/TIM.2015.2479101］
Lu S F and Liu Z. 2016. Automatic visual inspection of a missing split pin in the China railway high-speed. Applied Optics， 55（30）： 8395-8405 ［DOI： 10.1364/AO.55.008395http://dx.doi.org/10.1364/AO.55.008395］
Lu S F， Liu Z and Shen Y. 2018. Automatic fault detection of multiple targets in railway maintenance based on time-scale normalization. IEEE Transactions on Instrumentation and Measurement， 67（4）： 849-865 ［DOI： 10.1109/TIM.2018.2790498http://dx.doi.org/10.1109/TIM.2018.2790498］
Ma L F， Luo F， Yan J P， Xu Z， Luo J and Li X. 2021. Deep-learning based medical image registration pathway： towards unsupervised learning. Journal of Image and Graphics， 26（9）： 2037-2057
马露凡， 罗凤， 严江鹏， 徐哲， 罗捷， 李秀. 2021. 深度医学图像配准研究进展： 迈向无监督学习. 中国图象图形学报， 26（9）： 2037-2057 ［DOI： 10.11834/jig.200361http://dx.doi.org/10.11834/jig.200361］
Murphy J M， Le Moigne J and Harding D J. 2016. Automatic image registration of multimodal remotely sensed data with global shearlet features. IEEE Transactions on Geoscience and Remote Sensing， 54（3）： 1685-1704 ［DOI： 10.1109/TGRS.2015.2487457http://dx.doi.org/10.1109/TGRS.2015.2487457］
Rocco I， Arandjelovic R and Sivic J. 2017. Convolutional neural network architecture for geometric matching//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu， USA： IEEE： 39-48 ［DOI： 10.1109/CVPR.2017.12http://dx.doi.org/10.1109/CVPR.2017.12］
Silveira G and Malis E. 2007. Real-time visual tracking under arbitrary illumination changes//Proceedings of 2007 IEEE Conference on Computer Vision and Pattern Recognition. Minneapolis， USA： IEEE： 1-6 ［DOI： 10.1109/CVPR.2007.382993http://dx.doi.org/10.1109/CVPR.2007.382993］
Steger C， Ulrich M and Wiedemann C. 2018. Machine Vision Algorithms and Applications. 2nd ed. Weinheim： Wiley-VCH
Storey J C， Choate M J and Meyer D J. 2004. A geometric performance assessment of the EO-1 advanced land imager. IEEE Transactions on Geoscience and Remote Sensing， 42（3）： 602-607 ［DOI： 10.1109/TGRS.2003.820603http://dx.doi.org/10.1109/TGRS.2003.820603］
Xu J Y， Sun R， Tian Y P， Xie Q， Yang Y， Liu H D and Cao L. 2015. Correction of rolling wheel images captured by a linear array camera. Applied Optics， 54（33）： 9736-9740 ［DOI： 10.1364/AO.54.009736http://dx.doi.org/10.1364/AO.54.009736］
Yang Z Q， Dan T T and Yang Y. 2018. Multi-temporal remote sensing image registration using deep convolutional features. IEEE Access， 6： 38544-38555 ［DOI： 10.1109/ACCESS.2018.2853100http://dx.doi.org/10.1109/ACCESS.2018.2853100］
Ye Y T， Xiao J， Rao J Z， Li J F， Yang C P， Yang G， Zhong J and Ao M W. 2011. Optical Tutorial. 2nd ed. Beijing： Tsinghua University Press
叶玉堂， 肖峻， 饶建珍，李剑峰， 杨春平， 杨刚， 钟建， 敖明武. 2011. 光学教程.2版. 北京： 清华大学出版社
Ye Y X， Shan J， Bruzzone L and Shen L. 2017. Robust registration of multimodal remote sensing images based on structural similarity. IEEE Transactions on Geoscience and Remote Sensing， 55（5）： 2941-2958 ［DOI： 10.1109/TGRS.2017.2656380http://dx.doi.org/10.1109/TGRS.2017.2656380］
Yin Q Y， Huang Y， Zhang J G， Wu S and Wang L. 2021. Survey on deep learning based cross-modal retrieval. Journal of Image and Graphics， 26（6）： 1368-1388
尹奇跃， 黄岩， 张俊格， 吴书， 王亮. 2021. 基于深度学习的跨模态检索综述. 中国图象图形学报， 26（6）： 1368-1388 ［DOI： 10.11834/jig.200862http://dx.doi.org/10.11834/jig.200862］
Zhang X X， Gilliam C and Blu T. 2020. All-pass parametric image registration. IEEE Transactions on Image Processing， 29： 5625-5640 ［DOI： 10.1109/TIP.2020.2984897http://dx.doi.org/10.1109/TIP.2020.2984897］
Zhao B， Dai M R， Li P， Ma X N and Wu Y H. 2019. Research on defect detection of railway key components based on deep learning. Journal of the China Railway Society， 41（8）： 67-73
赵冰， 代明睿， 李平， 马小宁， 吴艳华. 2019. 基于深度学习的铁路关键部件缺陷检测研究. 铁道学报， 41（8）： 67-73 ［DOI： 10.3969/j.issn.1001-8360.2019.08.009http://dx.doi.org/10.3969/j.issn.1001-8360.2019.08.009］
Zhou W， Shi T Y， Li P， Ma X N and Yang K. 2019. Defects detection and segmentation of operation safety image of EMU based on convolutional neural network. Journal of the China Railway Society， 41（10）： 76-83
周雯， 史天运， 李平， 马小宁， 杨凯. 2019. 基于卷积神经网络的动车组行车安全图像缺陷检测与分割. 铁道学报， 41（10）： 76-83 ［DOI： 10.3969/j.issn.1001-8360.2019.10.011http://dx.doi.org/10.3969/j.issn.1001-8360.2019.10.011］
相关作者
相关机构
京公网安备11010802024621