浏览全部资源
扫码关注微信
收稿日期：2024-06-13，
修回日期：2024-08-13，
纸质出版日期：2025-03-16
移动端阅览
目的
2
针对传统基于深度学习的去模糊方法存在的伪影明显、细节模糊和噪声残留等问题，提出一种基于提示学习的多尺度图像去模糊新方法。
方法
2
首先，在详细分析传统去模糊方法的基础上，引入基于提示学习的特定退化信息编码模块，利用退化信息中包含的上下文信息来动态地引导深度网络以更有效地完成去模糊任务。其次，设计了新的门控前馈网络，通过控制各个层级的信息流动构建更为丰富和更具层次结构的特征表示，从而进一步提高对复杂数据的理解和处理能力，以更好地保持结果图像的几何结构。另外，新方法引入了经典的总变差正则来抑制去模糊过程中的噪声残留，以提高结果图像的视觉表现。
结果
2
基于GoPro和REDS（the realistic and diverse scenes）数据集的大量实验结果表明，与其他先进的基于深度学习的去模糊方法相比，本文方法在图像去模糊方面取得了更好的效果。在峰值信噪比（peak signal-to-noise ratio， PSNR）和结构相似性（structural similarity， SSIM）指标上，本文方法在GoPro数据集上分别达到33.04 dB和0.962的最优结果。在REDS数据集上分别达到28.70 dB和0.859的结果。并且，相比SAM-deblur（segment anything model-deblur）方法，PSNR提升了1.77 dB。
结论
2
相较于其他的去模糊方法，本文方法不仅能够较好地保持结果图像的细节信息，而且还能够有效地克服伪影明显和噪声残留的问题，所得结果图像在PSNR和SSIM等客观评价指标方面均有更好的表现。
Objective
2
Image deblurring aims to restore a clean image from blurry images while still maintaining the structure and details of the original image during the restoration. With the rapid development of Internet technology， the way people obtain images becomes highly diversified. However， the image is often blurred or distorted by various factors during the acquisition process. Therefore， deblurring the image is necessary. Image deblurring is of considerable importance to improve image quality and plays a key role in numerous fields such as medical imaging， satellite image processing， and security monitoring， which has attracted the attention of many researchers. Additional prior knowledge is needed to recover images with high quality due to the ill-posed image deblurring task. At present， the existing deblurring methods include traditional and deep learning-based approaches. In the traditional methods， despite the simplicity and convenience of the filter-based deblurring method， the recovered images often have artifacts， content loss， and other problems， which fail to meet the needs of various applications. The deblurring method based on the idea of regularity has received increasing attention from researchers for a long time， and various methods of constructing regular terms have been proposed to solve this kind of ill-posed problems. These traditional methods can achieve the purpose of deblurring to a certain extent. However， they rely on the prior information of images， which is difficult to obtain accurately in practical applications. Therefore， this kind of method cannot be effectively promoted in a wide range. With the extensive application of deep learning technology， an increasing number of researchers begin to use this technology to address ill-posed problems. The image deblurring methods generally fall into three main categories： convolutional neural network （CNN）-based method， generative adversarial network （GAN）-based method， and Transformer-based method. In the CNN-based methods， the powerful feature extraction capability of CNNs allows the model to learn complex mapping relationships. By minimizing the loss function， these methods guide the model’s convergence to obtain the best output images. However， such approaches often lack multiscale features and can introduce artifacts and result in the loss of image details. Researchers have proposed a new framework named GAN to address these shortcomings. In this approach， the generator and discriminator are trained alternately to continuously improve the performance of the generator， leading to higher-quality output images. Following the success of Transformers in natural language processing， researchers have begun to introduce them into the field of image processing. The advantage of Transformer-based methods is their capability to better capture local context information， leading to improved image deblurring. However， incorporating Transformer blocks inevitably increases the computational complexity of the model. A novel multiscale image deblurring method based on prompt learning is proposed to address the problems of noticeable artifacts， fuzzy details and residual noise in previous image deblurring methods.
Method
2
In this paper， three improvements are made. First， the degraded information coding module based on Prompt learning can use the context information in the degraded image to dynamically guide the deep network to complete different image deblurring tasks. Next， a gated feedforward network is designed to control the flow of information at each level and build a richer and more hierarchical feature representation. Therefore， prompt U-shaped block （PUBlock） is designed. In addition， considering the original loss function， the adaptive total variation regularization is added to effectively suppress the noise residue in the process of image restoration and improve the visual performance of the result image. Through the introduction of a gating mechanism， the network can dynamically control the flow of information to effectively capture complex feature relationships. Using deep convolution can improve the efficiency of the model while ensuring its performance. Prompt learning can effectively help the model utilize degraded images， and adaptive regularization can selectively smooth the image， which not only removes the noise but also prevents the image from being over-smooth.
Result
2
Deblurring experiments are performed on the GoPro and REDS datasets and compared them with other advanced methods to demonstrate the effectiveness of the proposed method. In addition， peak signal-to-noise ratio （PSNR） and structural similarity （SSIM） are used as objective evaluation metrics. Experimental results show that the proposed method outperforms all other methods in GoPro and REDS datasets and achieves 33.04 dB and 0.962， respectively， on the GoPro dataset and 28.70 dB and 0.859， respectively， on the REDS dataset under the two metrics. These results are better than the PSNR and SSIM values of the conventional image deblurring method. The comparison results with the segment anything model-deblur （SAM-deblur） algorithm show that PSNR improves by 1.77 dB on the REDS dataset， while those with double-scale network with deep feature fusion attention （DFFA-Net） based on the GoPro dataset show that the proposed method improve the PSNR and SSIM by 0.49 dB and 0.005， respectively. In addition， the visual results reveal that the images recovered by the proposed model are closest to the original real image， maintaining the original structure and features， and exhibits a finer edge.
Conclusion
2
In this paper， aiming to address the problems of existing image deblurring methods， a novel multiscale image deblurring method based on prompt learning is introduced. The experimental results show that the new method not only preserves the details of the result image but also effectively overcomes the problems of evident artifacts and noise residue. The result image also has superior performance in the objective evaluation metrics on PSNR and SSIM.
Abdelhamed A ， Lin S and Brown M S . 2018 . A high-quality denoising dataset for smartphone cameras // Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Salt Lake City， USA ： IEEE： 1692 - 1700 ［ DOI： 10.1109/CVPR.2018.00182 http://dx.doi.org/10.1109/CVPR.2018.00182 ］
Ba J L ， Kiros J R and Hinton G E . 2016 . Layer normalization ［EB/OL］. ［ 2024-06-13 ］. https://arxiv.org/pdf/1607.06450.pdf https://arxiv.org/pdf/1607.06450.pdf
Buades A ， Coll B and Morel J M . 2006 . Image enhancement by non-local reverse heat equation . Preprint CMLA ， 22 ： #2006
Cai J R ， Zuo W M and Zhang L . 2020 . Dark and bright channel prior embedded network for dynamic scene deblurring . IEEE Transactions on Image Processing ， 29 ： 6885 - 6897 ［ DOI： 10.1109/TIP.2020.2995048 http://dx.doi.org/10.1109/TIP.2020.2995048 ］
Chang M ， Li Q ， Feng H J and Xu Z H . 2020 . Spatial-adaptive network for single image denoising // Proceedings of the 16th European Conference on Computer Vision . Glasgow， UK ： Springer： 171 - 187 ［ DOI： 10.1007/978-3-030-58577-8_11 http://dx.doi.org/10.1007/978-3-030-58577-8_11 ］
Chen J B ， Xiong B S ， Kuang F and Zhang Z Z . 2023 . Motion deblurring based on deep feature fusion attention and double-scale . Journal of Image and Graphics ， 28 （ 12 ）： 3731 - 3743
陈加保 ， 熊邦书 ， 况发 ， 章照中 . 2023 . 深度特征融合注意力与双尺度的运动去模糊 . 中国图象图形学报 ， 28 （ 12 ）： 3731 - 3743 ［ DOI： 10.11834/jig.220931 http://dx.doi.org/10.11834/jig.220931 ］
Chen L Y ， Chu X J ， Zhang X Y and Sun J . 2022 . Simple baselines for image restoration // Proceedings of the 17th European Conference on Computer Vision . Tel Aviv， Israel ： Springer： 17 - 33 ［ DOI： 10.1007/978-3-031-20071-7_2 http://dx.doi.org/10.1007/978-3-031-20071-7_2 ］
Cho S J ， Ji S W ， Hong J P ， Jung S W and Ko S J . 2021 . Rethinking coarse-to-fine approach in single image deblurring // Proceedings of 2021 IEEE/CVF International Conference on Computer Vision . Montreal， Canada ： IEEE： 4621 - 4630 ［ DOI： 10.1109/ICCV48922.2021.00460 http://dx.doi.org/10.1109/ICCV48922.2021.00460 ］
Erdem A T and Tekalp A M . 1990 . Blur identification using bispectrum // Proceedings of 1990 International Conference on Acoustics， Speech， and Signal Processing . Albuquerque， USA ： IEEE： 1961 - 1964 ［ DOI： 10.1109/ICASSP.1990.115892 http://dx.doi.org/10.1109/ICASSP.1990.115892 ］
Fan C M ， Liu T J ， Liu K H and Chiu C H . 2022 . Selective residual M-Net for real image denoising // Proceedings of the 30th European Signal Processing Conference （EUSIPCO） . Belgrade， Serbia ： IEEE： 469 - 473 ［ DOI： 10.23919/EUSIPCO55093.2022.9909521 http://dx.doi.org/10.23919/EUSIPCO55093.2022.9909521 ］
Jiang Z ， Zhang Y ， Zou D Q ， Ren J ， Lv J C and Liu Y B . 2020 . Learning event-based motion deblurring // Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Seattle， USA ： IEEE： 3317 - 3326 ［ DOI： 10.1109/CVPR42600.2020.00338 http://dx.doi.org/10.1109/CVPR42600.2020.00338 ］
Khare A and Tiwary U S . 2005 . A new method for deblurring and denoising of medical images using complex wavelet transform // Proceedings of 2005 IEEE Engineering in Medicine and Biology 27th Annual Conference . Shanghai， China ： IEEE： 1897 - 1900 ［ DOI： 10.1109/IEMBS.2005.1616821 http://dx.doi.org/10.1109/IEMBS.2005.1616821 ］
Kheradmand A and Milanfar P . 2014 . A general framework for regularized， similarity-based image restoration . IEEE Transactions on Image Processing ， 23 （ 12 ）： 5136 - 5151 ［ DOI： 10.1109/TIP.2014.2362059 http://dx.doi.org/10.1109/TIP.2014.2362059 ］
Köhler R ， Hirsch M ， Mohler B ， Schölkopf B and Harmeling S . 2012 . Recording and playback of camera shake： benchmarking blind deconvolution with a real-world database // Proceedings of the 12th European Conference on Computer Vision . Florence， Italy ： Springer： 27 - 40 ［ DOI： 10.1007/978-3-642-33786-4_3 http://dx.doi.org/10.1007/978-3-642-33786-4_3 ］
Kupyn O ， Budzan V ， Mykhailych M ， Mishkin D and Matas J . 2018 . DeblurGAN： blind motion deblurring using conditional adversarial networks // Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Salt Lake City， USA ： IEEE： 8183 - 8192 ［ DOI： 10.1109/CVPR.2018.00854 http://dx.doi.org/10.1109/CVPR.2018.00854 ］
Kupyn O ， Martyniuk T ， Wu J R and Wang Z Y . 2019 . DeblurGAN-v2： deblurring （orders-of-magnitude） faster and better // Proceedings of 2019 IEEE/CVF International Conference on Computer Vision . Seoul， Korea （South） ： IEEE： 8877 - 8886 ［ DOI： 10.1109/ICCV.2019.00897 http://dx.doi.org/10.1109/ICCV.2019.00897 ］
Levin A . 2007 . Blind motion deblurring using image statistics // Proceedings of the 19th International Conference on Neural Information Processing Systems . Canada ： MIT Press： 841 - 848
Li S W ， Liu M X ， Zhang Y T ， Chen S ， Li H X ， Dou Z F and Chen H . 2024 . SAM-DEBLUR： let segment anything boost image deblurring // Proceedings of 2024 IEEE International Conference on Acoustics， Speech and Signal Processing . Seoul， Korea （South） ： IEEE： 2445 - 2449 ［ DOI： 10.1109/ICASSP48485.2024.10445844 http://dx.doi.org/10.1109/ICASSP48485.2024.10445844 ］
Nah S ， Kim T H and Lee K M . 2017 . Deep multi-scale convolutional neural network for dynamic scene deblurring // Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition . Honolulu， USA ： IEEE： 257 - 265 ［ DOI： 10.1109/CVPR.2017.35 http://dx.doi.org/10.1109/CVPR.2017.35 ］
Nah S， Son S， Lee S， Timofte R， Lee K M， Chen L Y， Zhang J， Lu X， Chu X J， Chen C P， Xiong Z W， Xu R K， Xiao Z Y， Huang J， Zhang Y Y， Xi S， Wei J， Bai H R， Cheng S S， Wei H， Sun L， Tang J H， Pan J S， Lee D， Lee C， Kim T， Wang X B， Zhang D F， Pan Z H， Lin T W， Wu W H， He D L， Li B P， Li B Y， Xi T， Zhang G， Liu J T， Han J Y， Ding E R， Tao G P， Chu W Q， Cao Y， Luo D H， Tai Y， Lu T， Wang C J， Li J L， Huang F Y， Chen H T， Chen S J， Guo T Y， Wang Y H， Zamir S W， Arora A， Khan S， Hayat M， Khan F S， Shao L， Zuo Y S， Ou Y M， Chai Y J， Shi L， Liu S， Lei L， Feng C Y， Zeng K， Yao Y Y， Liu X R， Zhang Z Z， Huang H C， Zhang Y C， Jiang M C， Zou W B， Miao S， Kim Y， Sun Y J， Deng S Y， Ren W Q， Cao X C， Wang T， Suin M， Rajagopalan A N， Van Duong V， Nguyen T H， Yim J， Jeon B， Li R， Xie J W， Han J W， Choi J H， Kim J H， Lee J S， Zhang J X， Peng F， Svitov D， Pakulich D， Kim J and Jeong J. 2021 . NTIRE 2021 challenge on image deblurring//Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville， USA ： IEEE： 149 - 165 ［ DOI： 10.1109/CVPRW53098.2021.00025 http://dx.doi.org/10.1109/CVPRW53098.2021.00025 ］
Park D ， Kang D U ， Kim J and Chun S Y . 2020 . Multi-temporal recurrent neural networks for progressive non-uniform single image deblurring with incremental temporal training // Proceedings of the 16th European Conference on Computer Vision . Glasgow， UK ： Springer： 327 - 343 ［ DOI： 10.1007/978-3-030-58539-6_20 http://dx.doi.org/10.1007/978-3-030-58539-6_20 ］
Potlapalli V ， Zamir S W ， Khan S and Khan F S . 2023 . PromptiR： prompting for all-in-one blind image restoration ［EB/OL］. ［ 2023-06-22 ］. https://arxiv.org/pdf/2306.13090.pdf https://arxiv.org/pdf/2306.13090.pdf
Rudin L I ， Osher S and Fatemi E . 1992 . Nonlinear total variation based noise removal algorithms . Physica D ： Nonlinear Phenomena ， 60 （ 1/4 ）： 259 - 268 ［ DOI： 10.1016/0167-2789（92）90242-F http://dx.doi.org/10.1016/0167-2789（92）90242-F ］
Sun J ， Cao W F ， Xu Z B and Ponce J . 2015 . Learning a convolutional neural network for non-uniform motion blur removal // Proceedings of 2015 IEEE Conference on Computer Vision and Pattern Recognition . Boston， USA ： IEEE： 769 - 777 ［ DOI： 10.1109/CVPR.2015.7298677 http://dx.doi.org/10.1109/CVPR.2015.7298677 ］
Tao X ， Gao H Y ， Shen X Y ， Wang J and Jia J Y . 2018 . Scale-recurrent network for deep image deblurring // Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Salt Lake City， USA ： IEEE： 8174 - 8182 ［ DOI： 10.1109/CVPR.2018.00853 http://dx.doi.org/10.1109/CVPR.2018.00853 ］
Tu Z Z ， Talebi H ， Zhang H ， Yang F ， Milanfar P ， Bovik A and Li Y X . 2022 . MAXIM： multi-axis MLP for image processing // Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition . New Orleans， USA ： IEEE： 5759 - 5770 ［ DOI： 10.1109/CVPR52688.2022.00568 http://dx.doi.org/10.1109/CVPR52688.2022.00568 ］
Wang Z D ， Cun X ， Bao J ， Zhou W ， Liu J and Li H . 2022 . Uformer： a general U-shaped Transformer for image restoration // Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition . New Orleans， USA ： IEEE： 17662 - 17672 ［ DOI： 10.1109/CVPR52688.2022.01716 http://dx.doi.org/10.1109/CVPR52688.2022.01716 ］
Wang Z M and Bao H . 2013 . A new regularization model based on non-local means for image deblurring . Applied Mechanics and Materials ， 411 - 414 ： 1164 - 1169 ［ DOI： 10.4028/www.scientific.net/AMM.411-414.1164 http://dx.doi.org/10.4028/www.scientific.net/AMM.411-414.1164 ］
Yue Z S ， Zhao Q ， Zhang L and Meng D Y . 2020 . Dual adversarial network： toward real-world noise removal and noise generation // Proceedings of the 16th European Conference on Computer Vision . Glasgow， UK ： Springer： 41 - 58 ［ DOI： 10.1007/978-3-030-58607-2_3 http://dx.doi.org/10.1007/978-3-030-58607-2_3 ］
Zamir S W ， Arora A ， Khan S ， Hayat M ， Khan F S and Yang M H . 2022 . Restormer： efficient Transformer for high-resolution image restoration // Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition . New Orleans， USA ： IEEE： 5718 - 5729 ［ DOI： 10.1109/CVPR52688.2022.00564 http://dx.doi.org/10.1109/CVPR52688.2022.00564 ］
Zamir S W ， Arora A ， Khan S ， Hayat M ， Khan F S ， Yang M H and Shao L . 2020a . Learning enriched features for real image restoration and enhancement // Proceedings of the 16th European Conference on Computer Vision . Glasgow， UK ： Springer： 492 - 511 ［ DOI： 10.1007/978-3-030-58595-2_30 http://dx.doi.org/10.1007/978-3-030-58595-2_30 ］
Zamir S W ， Arora A ， Khan S ， Hayat M ， Khan F S ， Yang M H and Shao L . 2020b . CycleISP： real image restoration via improved data synthesis // Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Seattle， USA ： IEEE： 2693 - 2702 ［ DOI： 10.1109/CVPR42600.2020.00277 http://dx.doi.org/10.1109/CVPR42600.2020.00277 ］
Zamir S W ， Arora A ， Khan S ， Hayat M ， Khan F S ， Yang M H and Shao L . 2021 . Multi-stage progressive image restoration // Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Nashville， USA ： IEEE： 14816 - 14826 ［ DOI： 10.1109/CVPR46437.2021.01458 http://dx.doi.org/10.1109/CVPR46437.2021.01458 ］
Zhang H C ， Yang J C ， Zhang Y N and Huang T S . 2011 . Sparse representation based blind image deblurring // Proceedings of 2011 IEEE International Conference on Multimedia and Expo . Barcelona， Spain ： IEEE： 1 - 6 ［ DOI： 10.1109/ICME.2011.6012035 http://dx.doi.org/10.1109/ICME.2011.6012035 ］
Zhang H G ， Dai Y C ， Li H D and Koniusz P . 2019 . Deep stacked hierarchical multi-patch network for image deblurring // Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Long Beach， USA ： IEEE： 5971 - 5979 ［ DOI： 10.1109/CVPR.2019.00613 http://dx.doi.org/10.1109/CVPR.2019.00613 ］
Zhang J W ， Pan J S ， Ren J ， Song Y B ， Bao L C ， Lau R W H and Yang M H . 2018 . Dynamic scene deblurring using spatially variant recurrent neural networks // Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Salt Lake City， USA ： IEEE： 2521 - 2529 ［ DOI： 10.1109/CVPR.2018.00267 http://dx.doi.org/10.1109/CVPR.2018.00267 ］
Zhang K H ， Luo W H ， Zhong Y R ， Ma L ， Stenger B ， Liu W and Li H D . 2020 . Deblurring by realistic blurring // Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition . Seattle， USA ： IEEE： 2734 - 2743 ［ DOI： 10.1109/CVPR42600.2020.00281 http://dx.doi.org/10.1109/CVPR42600.2020.00281 ］
相关作者
相关机构
京公网安备11010802024621