浏览全部资源
扫码关注微信
网络出版日期： 2024-12-30 ，
移动端阅览
洪雨辰,吕游伟,万人杰等.基于深度学习的图像反射消除方法综述[J].中国图象图形学报,
Hong Yuchen,Lyu Youwei,Wan Renjie,et al.Deep learning for image reflection removal： a survey[J].Journal of Image and Graphics,
随着智能手机摄影的普及，图像数据采集变得极为便捷，但在透过透明介质例如玻璃窗进行拍摄时，玻璃反射的存在严重影响了图像质量，进而干扰下游计算机视觉任务的性能。反射消除作为计算摄像学与计算机视觉领域的重要研究问题，旨在从带反射图像中消除反射干扰以恢复清晰的背景图像。随着深度学习在计算摄像问题中的广泛应用，反射消除领域经历了快速发展，鉴于此，本文旨在围绕近年来基于深度学习的反射消除研究进展进行深入探讨。首先，从混合图像成像模型入手，分析玻璃材质特性以及相机特性对反射图像和背景图像性质的影响。其次，从输入图像的角度，详细汇总了现有的反射消除真实数据集，并对其应用场景、具体用途、数据规模和分辨率等属性进行了统计分析。接着，从深度学习模型的视角，系统性对比了反射消除网络的设计范式、损失函数和评估指标。此外，根据反射消除方法所依赖的分层依据和辅助信息，将现有方法归纳为基于图像特征、文本特征、几何特性和光照特性四大类，并进行了简明的描述和分析。最后，通过讨论反射消除领域内尚未解决的关键挑战，对该领域进行总结与展望。本文旨在提供一个关于反射消除问题的系统研究视角，帮助研究者建立对反射消除技术的深刻认识，为未来研究提供有价值的参考。
With the widespread adoption of smartphone photography， the acquisition of image data has become extremely convenient， generating massive amounts of image data that support the training of intelligent visual perception models. However， there exists a series of image degradation issues that hinder the leverage of captured image data， one of which is the glass reflection. When people take photos through transparent materials such as glass windows， the presence of reflections can severely degrade the quality of the captured images and interfere with downstream computer vision tasks. Reflection removal aims at separating different scene components located on either side of the glass from reflection-contaminated mixture images， thereby eliminating glass reflections to obtain clear transmission images. As an attractive topic in computational photography and computer vision， reflection removal has garnered significant attention from researchers and experienced rapid development with the extensive application of deep learning in computational photography problems. This paper comprehensively reviews recent advancements in deep-learning-based reflection removal. Firstly， we start by analyzing the image formation model of mixture images， examining the effects of glass material and camera characteristics on the properties of reflection and transmission images， including refraction， absorption， and reflection effects of glass and image blur caused by camera depth of field. Secondly， from the perspective of input images， we summarize existing reflection removal datasets and conduct statistical analysis of their application scenarios， specific purposes， data scale， and resolution attributes. Synthetic data created based on theoretical imaging models are important for large-scale training dataset creation， but they still exhibit discrepancies compared to the distribution of real captured images. Therefore， a key strategy to enhance model performance is to use a portion of real data during training. To comprehensively evaluate the performance of reflection removal algorithms in real-world settings， it is essential to construct benchmark datasets from real data. Thirdly， from the viewpoint of deep learning models， we systematically compare the network design， loss functions， and evaluation metrics of reflection removal networks. For network design， according to different strategies to predict transmission and reflection images， researchers primarily employ three types of network structures to construct the network models： direct， cascaded， and concurrent structures. As for loss functions， early methods mainly utilized pixel loss and edge loss. Subsequently， more sophisticated loss functions were introduced to constrain the perceptual quality of predicted images and the correlation between transmission and reflection images， guiding the optimization of network models toward more realistic and higher-quality restoration. For evaluating the quality of reflection removal results， similarities across various statistical characteristics between the predicted images and the reference images are used as metrics. Based on the employed auxiliary information in reflection removal methods， we propose a systematical taxonomy to categorize existing approaches into four types： image feature-based， text feature-based， geometry characteristics-based， and light characteristics-based. With the rapid development of deep learning， a series of reflection removal methods based on image features have utilized deep neural networks to extract low-level or high-level image features from large training datasets to facilitate reflection removal. However， due to the inherent ill-posed nature of the problem， introducing additional auxiliary information becomes crucial when dealing with complex reflection scenarios. Methods based on geometry characteristics use panoramic cameras or capture multiple images from different camera positions to obtain additional views of scenes， providing auxiliary contextual information. Methods based on light characteristics leverage the discrepancy in the light paths of rays from the transmission and reflection scenes， such as variations in scene lighting conditions or polarization characteristics， to provide key clues for reflection removal. With the rise of multimodal large language models， methods based on text features introduce natural language descriptions to cooperate with the image modality and provide semantic guidance for reflection removal， achieving state-of-the-art results without requiring additional hardware support. Finally， by discussing unresolved key challenges within the field， we offer a summary and outlook for reflection removal research. This survey provides a systematic review on recent advances in deep learning for the reflection removal problem， which helps researchers to develop a profound understanding of reflection removal techniques and facilitate future research.
计算摄像学图像复原反射消除卷积神经网络扩散模型感知质量
Computational photographyImage restorationReflection removalConvolutional neural networkDiffusion modelPerceptual quality
Chang Y K， Jung C， Sun J， and Wang F Q. 2020. Siamese dense net- work for reflection removal with flash and no-flash image pairs. International Journal of Computer Vision， 128（6）： 1673–1698 ［DOI： 10.1007/S11263-019-01276-Zhttp://dx.doi.org/10.1007/S11263-019-01276-Z］
Chugunov I， Shustin D， Yan R Y， Lei C Y， and Heide F. 2024. Neural spline fields for burst image fusion and layer separation//Proceedings of Computer Vision and Pattern Recognition. IEEE： 25763–25773 ［DOI： 10.48550/ARXIV.2312.14235http://dx.doi.org/10.48550/ARXIV.2312.14235］
Dong Z， Xu K， Yang Y， Bao H J， Xu W W， and Lau R WH. 2021. Location-aware single image reflection removal//Proceedings of International Conference on Computer Vision. IEEE： 4997–5006 ［DOI： 10.1109/ICCV48922.2021.00497］.
Fan Q N， Yang J L， Hua G， Chen B Q， and Wipf D. 2017. A generic deep architecture for single image reflection removal and image smoothing//Proceedings of International Conference on Computer Vision. IEEE： 3258–3267 ［DOI： 10.1109/ICCV.2017.351http://dx.doi.org/10.1109/ICCV.2017.351］
Goodfellow I， Pouget-A J， Mirza M， Xu B， Warde-F D， Ozair S， Courville A， and Bengio Y. 2014. Generative adversarial nets//Proceedings of Advances in Neural Information Processing Systems. MIT Press： 2672–2680 ［DOI： 10.5555/2969033.2969125http://dx.doi.org/10.5555/2969033.2969125］
Han B J， and Sim J Y. 2022. Zero-Shot Learning for reflection removal of single 360-degree image//Proceedings of European Conference on Computer Vision. Springer： 533–548 ［DOI： 10.1007/978-3-031-19800-7\_31］
Hong Y C， Chang Y K， Liang J X， Ma L， Huang T J， and Shi B X. 2024a. Light flickering guided reflection removal. International Journal of Computer Vision， 132： 3933–3953 ［DOI： 10.1007/s11263-024-02073-zhttp://dx.doi.org/10.1007/s11263-024-02073-z］
Hong Y C， Lyu Y W， Li S， Cao G， and Shi B X. 2023a. Reflection removal with NIR and RGB image feature fusion. IEEE Transactions on Multimedia， 25： 7101–7112 ［DOI： 10.1109/TMM.2022.3217446http://dx.doi.org/10.1109/TMM.2022.3217446］
Hong Y C， Lyu Y W， Li S， and Shi B X. 2020. Near-infrared image guided reflection removal//Proceedings of International Conference on Multimedia and Expo. IEEE： 1–6 ［DOI： 10.1109/ICME46284.2020.9102937http://dx.doi.org/10.1109/ICME46284.2020.9102937］
Hong Y C， Zheng Q， Zhao L R， Jiang X D， Kot A C. and Shi B X. 2021. Panoramic image reflection removal//Proceedings of Computer Vision and Pattern Recognition. CVF/IEEE： 7762–7771 ［DOI： 10.1109/CVPR46437.2021.00767http://dx.doi.org/10.1109/CVPR46437.2021.00767］
Hong Y C， Zheng Q， Zhao L R， Jiang X D， Kot A C. and Shi B X. 2023b. PAR2Net： End-to-end panoramic image reflection removal. IEEE Transactions on Pattern Analysis and Machine Intelligence， 45（10）： 12192–12205 ［DOI： 10.1109/TPAMI.2023.3286429http://dx.doi.org/10.1109/TPAMI.2023.3286429］
Hong Y C， Zhong H F， Weng S C， Liang J X， and Shi B X. 2024b. L-DiffER： Single image reflection removal with language-based diffusion model//Proceedings of European Conference on Computer Vision. Springer
Hu Q M， and Guo X J. 2023. Single image reflection separation via component synergy//Proceedings of International Conference on Computer Vision. IEEE： 13092–13101 ［DOI： 10.1109/ICCV51070.2023.01208http://dx.doi.org/10.1109/ICCV51070.2023.01208］
Hu Q M， and Guo X J. 2021. Trash or treasure？ An interactive dual-stream strategy for single image reflection separation//Proceedings of Advances in Neural Information Processing Systems， Curran Associates Inc.： 24683–24694 ［DOI： 10.5555/3540261.3542151http://dx.doi.org/10.5555/3540261.3542151］
Jolicoeur-Martineau A. 2019. The relativistic discriminator： a key element missing from standard GAN//Proceedings of International Conference on Learning Representations. ［DOI： arXiv：1807.00734http://dx.doi.org/arXiv：1807.00734］
Kim S， Huo Y， and Yoon S E. 2020. Single image reflection removal with physically-based training images//Proceedings of Computer Vision and Pattern Recognition. CVF/IEEE： 5163–5172 ［DOI： 10.1109/CVPR42600.2020.00521http://dx.doi.org/10.1109/CVPR42600.2020.00521］
Lei C Y， and Chen Q F. 2021. Robust reflection removal with reflection-free flash-only cues//Proceedings of Computer Vision and Pattern Recognition. CVF/IEEE： 14811–14820 ［DOI： 10.1109/CVPR46437.2021.01457http://dx.doi.org/10.1109/CVPR46437.2021.01457］
Lei C Y， Huang X H， Zhang M D， Yan Q， Sun W X， and Chen Q F. 2020. Polarized reflection removal with perfect alignment in the wild//Proceedings of Computer Vision and Pattern Recognition. CVF/IEEE： 1747–1755 ［DOI： 10.1109/CVPR42600.2020.00182http://dx.doi.org/10.1109/CVPR42600.2020.00182］
Lei C Y， Jiang X D， and Chen Q F. 2023. Robust reflection removal with flash-only cues in the wild. IEEE Transactions on Pattern Analysis and Machine Intelligence， 45（12）： 15530–15545 ［DOI： 10.1109/TPAMI.2023.3314972http://dx.doi.org/10.1109/TPAMI.2023.3314972］
Levin A， and Weiss Y. 2007. User assisted separation of reflections from a single image using a sparsity prior. IEEE Transactions on Pattern Analysis and Machine Intelligence， 29（9）： 1647–1654 ［DOI： 10.1109/TPAMI.2007.1106http://dx.doi.org/10.1109/TPAMI.2007.1106］.
Li C， Yang Y X， He K， Lin S， and Hopcroft J E. 2020. Single image reflection removal through cascaded refinement//Proceedings of Computer Vision and Pattern Recognition. CVF/IEEE： 3562–3571 ［DOI： 10.1109/CVPR42600.2020.00362http://dx.doi.org/10.1109/CVPR42600.2020.00362］
Li R， Qiu S M， Zang G M， and Heidrich W. 2020. Reflection separation via multi-bounce polarization state tracing//Proceedings of European Conference on Computer Vision. Springer： 781–796 ［DOI： 10.1007/978-3-030-58601-0\_46］
Li Y， and Brown M S. 2013. Exploiting reflection change for automatic reflection removal//Proceedings of International Conference on Computer Vision. IEEE： 2432–2439 ［DOI： 10.1109/ICCV.2013.302http://dx.doi.org/10.1109/ICCV.2013.302］
Liu Y L， Lai W S， Yang M H， Chuang Y Y， and Huang J B. 2020. Learning to see through obstructions//Proceedings of Computer Vision and Pattern Recognition. CVF/IEEE： 14203–14212 ［DOI： 10.1109/CVPR42600.2020.01422http://dx.doi.org/10.1109/CVPR42600.2020.01422］
Liu Y L， Lai W S， Yang M H， Chuang Y Y， and Huang J B. 2021. Learning to see through obstructions with layered decomposition. IEEE Transactions on Pattern Analysis and Machine Intelligence， 44（11）： 8387–8402 ［DOI： 10.1109/TPAMI.2021.3111847http://dx.doi.org/10.1109/TPAMI.2021.3111847］
Lyu Y W， Cui Z P， Li S， Pollefeys M， and Shi B X. 2022. Physics-guided reflection separation from a pair of unpolarized and polarized images. IEEE Transactions on Pattern Analysis and Machine Intelligence， 45（2）： 2151–2165 ［DOI： 10.1109/TPAMI.2022.3162716http://dx.doi.org/10.1109/TPAMI.2022.3162716］
Lyu Y W， Cui Z P， Li S， Pollefeys M， and Shi B X. 2019. Reflection separation using a pair of unpolarized and polarized images//Proceedings of Advances in Neural Information Processing Systems. Curran Associates Inc.： 14532–14542 ［DOI： 10.5555/3454287.3455591http://dx.doi.org/10.5555/3454287.3455591］
Ma D Q， Wan R J， Shi B X， Kot A C， and Duan L Y. 2019. Learning to jointly generate and separate reflections//Proceedings of International Conference on Computer Vision. IEEE： 2444–2452 ［DOI： 10.1109/ICCV.2019.00253http://dx.doi.org/10.1109/ICCV.2019.00253］
Nam S， Brubaker M A. and Brown M S. 2022. Neural image representations for multi-image fusion and layer separation//Proceedings of European Conference on Computer Vision. Springer： 216–232 ［DOI： 10.1007/978-3-031-20071-7\_13］
Niklaus S， Zhang X C， Barron J T. Wadhwa N， Garg R， Liu F， and Xue T F. 2021. Learned dual-view reflection removal//Proceedings of Winter Conference on Applications of Computer Vision. IEEE： 3712–3721 ［DOI： 10.1109/WACV48630.2021.00376http://dx.doi.org/10.1109/WACV48630.2021.00376］
Park J， Kim H， Park E， and Sim J Y. 2024. Fully-automatic reflection removal for 360-degree images//Proceedings of Winter Conference on Applications of Computer Vision. IEEE： 1598–1606 ［DOI： 10.1109/WACV57701.2024.00163http://dx.doi.org/10.1109/WACV57701.2024.00163］
Ping J M， Liu Y， and Weng D D. 2021. Review of depth perception in virtual and real fusion environment. Journal of Image and Graphics， 26（6）：1503-1520
平佳敏， 刘越， 翁冬冬. 2021. 虚实融合场景中的深度感知研究综述. 中国图象图形学报， 26（6）：1503-1520 ［DOI： 10.11834/jig.210027http://dx.doi.org/10.11834/jig.210027］
Prasad B H P， S G R K. B Lokesh R. Mitra K. 2023. Burst reflection removal using reflection motion aggregation cues//Proceedings of Winter Conference on Applications of Computer Vision. IEEE： 239–248 ［DOI： 10.1109/WACV56688.2023.00032http://dx.doi.org/10.1109/WACV56688.2023.00032］
Prasad B H P， S G R K. B Lokesh R. Mitra K， and Chowdhury S. 2021. V-DESIRR： Very fast deep embedded single image reflection removal//Proceedings of International Conference on Computer Vision. IEEE： 2370–2379 ［DOI： 10.1109/ICCV48922.2021.00239http://dx.doi.org/10.1109/ICCV48922.2021.00239］
Shih Y C， Krishnan D， Durand F， and Freeman W T. 2015. Reflection removal using ghosting cues//Proceedings of Computer Vision and Pattern Recognition. IEEE： 3193–3201 ［DOI： 10.1109/CVPR.2015.7298939http://dx.doi.org/10.1109/CVPR.2015.7298939］
Simonyan K， and Zisserman A. 2014. Very deep convolutional networks for large-scale image recognition［EB/OL］.［2015-04-10］. https://arxiv.org/pdf/1409.1556https://arxiv.org/pdf/1409.1556
Sang G L， Xiao S D， and Zhao Q J. 2023. Soft threshold denoising and video data fusion-relevant low-quality 3D face recognition. Journal of Image and Graphics， 28（05）：1434-1444
桑高丽， 肖述笛， 赵启军. 2023. 联合软阈值去噪和视频数据融合的低质量3维人脸识别. 中国图象图形学报， 28（05）：1434-1444 ［DOI： 10.11834/jig.220695http://dx.doi.org/10.11834/jig.220695］
Wan R J， Shi B X， Duan L Y， Tan A H， and Kot A C. 2017. Benchmarking single-image reflection removal algorithms//Proceedings of International Conference on Computer Vision. IEEE： 3942–3950 ［DOI： 10.1109/ICCV.2017.423http://dx.doi.org/10.1109/ICCV.2017.423］
Wan R J， Shi B X， Duan L Y， Tan A H， and Kot A C. 2018. CRRN： Multi-scale guided concurrent reflection removal network//Proceedings of Computer Vision and Pattern Recognition. CVF/IEEE： 4777–4785 ［DOI： 10.1109/CVPR.2018.00502http://dx.doi.org/10.1109/CVPR.2018.00502］
Wan R J， Shi B X， Tan A H， and Kot A C. 2016. Depth of field guided reflection removal//Proceedings of International Conference on Image Processing. IEEE： 21–25 ［DOI： 10.1109/ICIP.2016.7532311http://dx.doi.org/10.1109/ICIP.2016.7532311］
Wan R J， Shi B X， Li H L， Duan L Y， and Kot A C. 2021. Face image reflection removal. International Journal of Computer Vision， 129（2）： 385–399 ［DOI： 10.1007/S11263-020-01372-5http://dx.doi.org/10.1007/S11263-020-01372-5］
Wan R J， Shi B X， Li H L， Duan L Y， Tan A H， and Kot A C. 2019. CoRRN： cooperative reflection removal network. IEEE Transactions on Pattern Analysis and Machine Intelligence， 42（12）： 2969–2982 ［DOI： 10.1109/TPAMI.2019.2921574http://dx.doi.org/10.1109/TPAMI.2019.2921574］
Wan R J， Shi B X， Li H L， Hong Y C， Duan L Y， and Kot A C. 2022. Benchmarking single-image reflection removal algorithms. IEEE Transactions on Pattern Analysis and Machine Intelligence， 45（2）： 1424–1441 ［DOI： 10.1109/TPAMI.2022.3168560http://dx.doi.org/10.1109/TPAMI.2022.3168560］
Wang C， Xu D J， Wan R J， He B， Shi B X， and Duan L Y. 2023. Background scene recovery from an image looking through colored glass. IEEE Transactions on Multimedia， 25： 2876–2887 ［DOI： 10.1109/TMM.2022.3152390http://dx.doi.org/10.1109/TMM.2022.3152390］
Wei K X， Yang J L， Fu Y， Wipf D， and Huang H. 2019. Single image reflection removal exploiting misaligned training data and network enhancements//Proceedings of Computer Vision and Pattern Recognition. CVF/IEEE： 8178–8187 ［DOI： 10.1109/CVPR.2019.00837http://dx.doi.org/10.1109/CVPR.2019.00837］
Wen Q， Tan Y J， Qin J， Liu W X， Han G Q， and He S F. 2019. Single image reflection removal beyond linearity//Proceedings of Computer Vision and Pattern Recognition. CVF/IEEE： 3771–3779 ［DOI： 10.1109/CVPR.2019.00389http://dx.doi.org/10.1109/CVPR.2019.00389］
Wieschollek P， Gallo O， Gu J W， and Kautz J. 2018. Separating reflection and transmission images in the wild//Proceedings of European Conference on Computer Vision. Springer： 90–105 ［DOI： 10.1007/978-3-030-01261-8\_6］
Xue T F， Rubinstein M， Liu C， and Freeman W T. 2015. A computational approach for obstruction-free photography. ACM Transactions on Graphics， 34（4）： 79：1–79：11 ［DOI： 10.1145/2766940http://dx.doi.org/10.1145/2766940］
Yue L， Tan H， Huang J K， and Zhang S K. Review of digital 3D indoor scene synthesis. Journal of Image and Graphics， 29（09）：2471-2493
岳亮， 谈皓， 黄俊凯， 张少魁. 2024. 数字室内三维场景构建综述. 中国图象图形学报， 29（09）：2471-2493 ［DOI： 10.11834/jig.230712http://dx.doi.org/10.11834/jig.230712］
Zhang Y N， Shen L L， and Li Q F. 2022. Content and gradient model-driven deep network for single image reflection removal//Proceedings of ACM International Conference on Multimedia. ACM： 6802–6812 ［DOI： 10.1145/3503161.3547918http://dx.doi.org/10.1145/3503161.3547918］
Zhang X E， Ng R， and Chen Q F. 2018. Single image reflection separation with perceptual losses//Proceedings of Computer Vision and Pattern Recognition. CVF/IEEE： 4786–4794 ［DOI： 10.1109/CVPR.2018.00503http://dx.doi.org/10.1109/CVPR.2018.00503］
Zhao H， Gallo O， Frosio I， and Kautz J. 2016. Loss functions for image restoration with neural networks. IEEE Transactions on computational imaging， 3（1）： 47–57 ［DOI： 10.1109/TCI.2016.2644865http://dx.doi.org/10.1109/TCI.2016.2644865］
Zheng Q， Chen J N， Lu Z， Shi B X， Jiang X D， Yap K H， Duan L Y， and Kot A C. 2020. What does plate glass reveal about camera calibration？//Proceedings of Computer Vision and Pattern Recognition. CVF/IEEE： 3019–3029 ［DOI： 10.1109/CVPR42600.2020.00309http://dx.doi.org/10.1109/CVPR42600.2020.00309］
Zheng Q， Shi B X， Chen J N， Jiang X D， Duan L Y， and Kot A C. 2021. Single image reflection removal with absorption effect//Proceedings of Computer Vision and Pattern Recognition. CVF/IEEE： 13395–13404 ［DOI： 10.1109/CVPR46437.2021.01319http://dx.doi.org/10.1109/CVPR46437.2021.01319］
Zhong H F， Hong Y C， Weng S C， Liang J X， and Shi B X. 2024. Language-guided image reflection separation//Proceedings of Computer Vision and Pattern Recognition. IEEE ［DOI： 10.48550/ARXIV.2402.11874http://dx.doi.org/10.48550/ARXIV.2402.11874］
Zhu Y R， Fu X Y， Jiang P T， Zhang H， Sun Q B， Chen J W， Zha Z J， and Li B. 2024. Revisiting single lmage reflection removal in the wild//Proceedings of Computer Vision and Pattern Recognition. IEEE ［DOI： 10.48550/arXiv.2311.17320http://dx.doi.org/10.48550/arXiv.2311.17320］
相关作者
相关机构
京公网安备11010802024621