浏览全部资源
扫码关注微信
纸质出版日期： 2022-03-16 ，
录用日期： 2021-11-01
移动端阅览
李书林, 冯朝路, 于鲲, 刘鑫, 江鑫, 赵大哲. 基于深度学习的心脏磁共振影像超分辨率前沿进展[J]. 中国图象图形学报, 2022,27(3):704-721.
Shulin Li, Chaolu Feng, Kun Yu, Xin Liu, Xin Jiang, Dazhe Zhao. Critical review of human cardiac magnetic resonance image super resolution reconstruction based on deep learning method[J]. Journal of Image and Graphics, 2022,27(3):704-721.
心脏为人体血液流动提供动力，是人体血液循环系统的重要组成部分。受人口老龄化影响，心脏病诊疗已成为重大公共健康话题。非侵入式活体心脏成像对心脏疾病的检测、诊断与治疗意义重大。然而，受活体心跳影响，成像扫描时间与心脏影像分辨率成为难以调和的矛盾。为缓和这一矛盾，基于快速扫描获得的低分辨率影像重建出心脏高分辨率影像的超分辨率(super-resolution
SR)重建技术成为研究热点。深度学习技术在医学影像处理领域中展现出强大生命力，基于深度学习的SR技术因其强大的学习能力与数据驱动性，在心脏影像SR重建领域中表现出明显优于传统方法的性能。目前领域内前沿成果较多，但缺少对领域现状进行总结、对未来发展进行展望的综述性文献。因此，本文对领域内现状进行梳理总结，挑选出代表性方法，分析方法特性，总结文献中心脏影像数据来源与规模，给出常用的评价指标，以及模型得出的性能评价结论。分析发现，基于深度学习的心脏SR重建技术取得了较大进展，但在运动伪影抑制、模型简化程度与时间性能方面仍有进步空间。此外，现有模型基本完全依靠网络强大的表达能力，鲜有临床先验知识的引入。最后，模型间性能对比相对较少，且领域内缺少代表性的可用于评价不同心脏SR重建模型性能的数据集。基于深度学习的心脏影像SR技术仍有较大发展空间。
Heart diseases diagnosis and treatment has become one of the major public health issues for human beings where non-invasive cardiac imaging is of great significance. Unfortunately
imaging scan time and cardiac image resolution become an intensive contradiction due to the characteristics of living heartbeat. Super-resolution (SR) image reconstruction like low-resolution-images based high-resolution cardiac images reconstruction are obtained based on the determined tolerable error within a short imaging time. Deep learning methods have revealed great vitality in the field of medical image processing nowadays. In terms of its good learning ability and data-driven factors
deep-learning-based SR reconstruction is qualified based on deep residual networks
generative adversarial networks compared with traditional methods. Our research analysis reviews the field via analyzing characteristics of representative methods
summing up cardiac image resources and the scale
summarizing commonly used evaluation indicators
giving performance evaluation and application conclusions of the methods and discussing methods in other fields that can be adapted for SR reconstructing cardiac magnetic resonance (CMR) images. Our analysis is derived from 13 opted literature reviews from Google
database systems and logic programming (DBLP)
and CNKI taking deep learning
cardiac and SR reconstruction as search keywords. This review first categorizes all methods by their evaluation datasets
which are open resource or not. Details on the datasets and links to the open-source datasets are also facilitated. By defining standard evaluation indicators based on the reconstructed SR images and their ground truth
namely high-resolution (HR) images
the performance of SR reconstruction methods can be evaluated quantitatively. This demonstration summarizes the 8 evaluation indicators in the cardiac SR reconstruction methods
including structural similarity
cardiovascular diameter measured value
dice coefficient. The evaluation indicators are divided into three categories with respect to the following aspects
those are evaluation of the quality of SR reconstructed images
evaluation of cardiac function and evaluation of the effectiveness of cardiac segmentation. Meanwhile
all 13 literature reviews are only used to increase the spatial resolution of the CMR images. Our research classifies the methods as CMR 2D SR reconstruction
CMR 3D SR reconstruction
and CMR SR reconstruction in other dimensions
depending on the dimension of the cardiac images processing based on the deep learning methods. In general
most CMR 2D SR reconstruction methods can reconstruct high resolution cardiac images in a relative short time span to assure SR reconstruction quality. In contrast
the CMR 3D SR reconstruction methods are involved 3D convolution
which take the spatial structure of the heart into account. The integrated information is analyzed amongst adjacent slices of CMR. Some of these methods achieve the SR reconstruction result qualified than CMR 2D SR reconstruction methods. However
a number of CMR data is small size and most of them are not open. The larger perceptual domain of the methods increases the computation complexity and reduces the temporal performance to some extent. As for CMR SR reconstruction in higher dimensions
corresponding methods meet the requirement of high-resolution image generation and image denoising in clinical analysis. All the selected CMR SR reconstruction methods can also be organized in accordance with network models and high-resolution image degradation methods
such as U-Net
generative adversarial network and long short-term memory network from the aspect of network models. Fourier degradation and different interpolation degradation methods are from the aspect of degradation methods. SR reconstructed high-resolution images can accurately facilitate heart anatomy
blood flow evaluation and heart tissue segmentation level. This research also reviewed the feasibility of adapting other SR reconstruction methods for CMR SR reconstruction as they are currently proposed and applied to images of other in vivo tissues and structures. Our research also tries to find some of the SR reconstruction methods from the field of natural images computer vision to discuss the feasibility that adapting them to CMR SR reconstruction
such as channel attention mechanisms
video SR methods and SR of real scenes. In summary
SR reconstruction of CMR images has its distinctive features than SR reconstruction of natural images like more diverse and purposeful evaluation metrics constraint of local reconstruction quality and to the difficulties of getting training data. It is found that deep-learning-based cardiac SR reconstruction has concerned more in motion artifact suppression
model simplification
and time performance further. In addition
current methods basically rely on the powerful expression ability of the CNN(convolutional neural network)
and little clinical prior knowledge is melted to the network to guide its learning. Performance comparison between existing models is relatively less
and there is no representative image repository to evaluate performances of different cardiac SR reconstruction methods.
深度学习心脏医学影像超分辨率重建磁共振成像心脏影像数据集卷积神经网络高分辨率影像
deep learningcardiac imagingsuper-resolution reconstructionmagnetic resonance imagingdatasets of cardiac imagesconvolutional neural networkshigh-resolution images
Bai W J, Shi W Z, O'Regan D P, Tong T, Wang H Y, Jamil-Copley S, Peters N S and Rueckert D. 2013. A probabilistic patch-based label fusion model for multi-atlas segmentation with registration refinement: application to cardiac MR images. IEEE Transactions on Medical Imaging, 32(7): 1302-1315[DOI: 10.1109/TMI.2013.2256922]
Bai Z Y and Tao J Y. 2022. Super-resolution reconstruction of brain images using Pseudo-3D convolutional network. Computer-Aided Design and Computer Graphics, 34
柏正尧, 陶劲宇. 2022. 采用伪3D卷积网络的脑部MRI图像超分辨率重建. 计算机辅助设计与图形学学报, 34[DOI: 10.3724/SP.J.1089.2022.18793http://dx.doi.org/10.3724/SP.J.1089.2022.18793]
Basty N and Grau V. 2018. Super resolution of cardiac cine MRI sequences using deep learning//Proceedings of the 3rd International Workshop on Reconstruction and Analysis of Moving Body Organs. Granada, Spain: Springer: 22-31 [DOI: 10.1007/978-3-030-00946-5_3http://dx.doi.org/10.1007/978-3-030-00946-5_3]
Bell-Kligler S, Shocher A and Irani M. 2019. Blind super-resolution kernel estimation using an internal-GAN//Proceedings of the 33rd Conference on Neural Information Processing Systems. Vancouver, Canada: NeurIPS: 284-293
Bromley J, Guyon I, LeCun Y, Säckinger E and Shah R. 1993. Signature verification using a "Siamese" time delay neural network//Proceedings of the 6th International Conference on Neural Information Processing Systems. Denver, USA: Morgan Kaufmann: 737-744 [DOI: 10.5555/2987189.2987282http://dx.doi.org/10.5555/2987189.2987282]
Cai J, Meng Z B and Ho C M. 2020. Residual channel attention generative adversarial network for image super-resolution and noise reduction//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Seattle, USA: IEEE: 1852-1861[DOI: 10.1109/CVPRW50498.2020.00235http://dx.doi.org/10.1109/CVPRW50498.2020.00235]
Chartrand G, Cheng P M, Vorontsov E, Drozdzal M, Turcotte S, Pal C J, Kadoury S and Tang A. 2017. Deep learning: a primer for radiologists. Radiographics, 37(7): 2113-2131[DOI: 10.1148/rg.2017170077]
Chen Z, Guo X Q, Yang C, Ibragimov B and Yuan Y X. 2020. Joint spatial-wavelet dual-stream network for super-resolution//Proceedings of the 23rd International Conference on Medical Image Computing and Computer-Assisted Intervention. Lima, Peru: Springer: 184-193[DOI: 10.1007/978-3-030-59722-1_18http://dx.doi.org/10.1007/978-3-030-59722-1_18]
Dong C, Loy C C, He K M and Tang X O. 2016. Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(2): 295-307 [DOI: 10.1109/TPAMI.2015.2439281]
Ferdian E, Suinesiaputra A, Dubowitz D J, Zhao D, Wang A, Cowan B and Young A A. 2020. 4DFlowNet: super-resolution 4D flow MRI using deep learning and computational fluid dynamics. Frontiers in Physics, 8: #138 [DOI: 10.3389/fphy.2020.00138]
Gholipour A, Estroff J A and Warfield S K. 2010. Robust super-resolution volume reconstruction from slice acquisitions: application to fetal brain MRI. IEEE Transactions on Medical Imaging, 29(10): 1739-1758 [DOI: 10.1109/TMI.2010.2051680]
Giannakidis A, Oktay O, Keegan J, Spadotto V, Voges I, Smith G, Pierce I, Bai W, Rueckert D, Ernst S, Gatzoulis M A, Pennell D J, Babu-Narayan S and Firmin D N. 2017. Super-resolution reconstruction of late gadolinium cardiovascular magnetic resonance images using a residual convolutional neural network//Proceedings of the 25th Scientific Meeting of the International Society for Magnetic Resonance in Medicine. Honolulu, USA: [s. n.]
Greenspan H. 2009. Super-resolution in medical imaging. The Computer Journal, 52(1): 43-63 [DOI: 10.1093/comjnl/bxm075]
Harris J L. 1964. Diffraction and resolving power. Optical Society of America, 54: 931-936[DOI:10.1364/JOSA.54.000931]
Hu J, Shen L and Sun G. 2018. Squeeze-and-excitation networks//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA: IEEE: 7132-7141[DOI: 10.1109/CVPR.2018.00745http://dx.doi.org/10.1109/CVPR.2018.00745]
Ji X Z, Cao Y, Tai Y, Wang C J, Li J L and Huang F Y. 2020. Real-world super-resolution via kernel estimation and noise injection//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Seattle, USA: IEEE: 1914-1923[DOI: 10.1109/CVPRW50498.2020.00241http://dx.doi.org/10.1109/CVPRW50498.2020.00241]
Kobayashi H, Nakayama R, Hizukuri A, Ishida M, Kitagawa K and Sakuma H. 2020. Improving image resolution of whole-heart coronary MRA using convolutional neural network. Journal of Digital Imaging, 33(2): 497-503 [DOI: 10.1007/s10278-019-00264-6]
Ledig C, Theis L, Huszár F, Caballero J, Cunningham A, Acosta A, Aitken A, Tejani A, Totz J, Wang Z H and Shi W Z. 2017. Photo-realistic single image super-resolution using a generative adversarial network//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, USA: IEEE: 105-114 [DOI: 10.1109/CVPR.2017.19http://dx.doi.org/10.1109/CVPR.2017.19]
Lim B, Son S, Kim H, Nah S and Lee K M. 2017. Enhanced deep residual networks for single image super-resolution//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops. Honolulu, USA: IEEE: 1132-1140 [DOI: 10.1109/CVPRW.2017.151http://dx.doi.org/10.1109/CVPRW.2017.151]
Lin J Y, Chang Y C and Hsu W H. 2020. Efficient and phase-aware video super-resolution for cardiac MRI//Proceedings of the 23rd International Conference on Medical Image Computing and Computer-Assisted Intervention. Lima, Peru: Springer: 66-76[DOI: 10.1007/978-3-030-59719-1_7http://dx.doi.org/10.1007/978-3-030-59719-1_7]
Mahapatra D and Bozorgtabar B. 2019. Progressive generative adversarial networks for medical image super resolution[EB/OL]. [2021-02-27].https://arxiv.org/pdf/1902.02144.pdfhttps://arxiv.org/pdf/1902.02144.pdf
Masutani E M, Bahrami N and Hsiao A. 2020. Deep learning single-frame and multiframe super-resolution for cardiac MRI. Radiology, 295(3): 552-561 [DOI: 10.1148/radiol.2020192173]
Mei Y Q, Fan Y C, Zhou Y Q, Huang L C, Huang T. S and Shi H H. 2020. Image super-resolution with cross-scale non-local attention and exhaustive self-exemplars mining//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, USA: IEEE: 5689-5698[DOI: 10.1109/CVPR42600.2020.00573http://dx.doi.org/10.1109/CVPR42600.2020.00573]
Oktay O, Bai W J, Lee M, Guerrero R, Kamnitsas K, Caballero J, de Marvao A, Cook S, O'Regan D and Rueckert D. 2016. Multi-input cardiac image super-resolution using convolutional neural networks//Proceedings of the 19th International Conference on Medical Image Computing and Computer-Assisted Intervention. Athens, Greece: Springer: 246-254[DOI: 10.1007/978-3-319-46726-9_29http://dx.doi.org/10.1007/978-3-319-46726-9_29]
Oktay O, Ferrante E, Kamnitsas K, Heinrich M, Bai W J, Caballero J, Cook S A, de Marvao A, Dawes T, O'Regan D P, Kainz B, Glocker B and Rueckert D. 2018. Anatomically constrained neural networks (ACNNs): application to cardiac image enhancement and segmentation. IEEE Transactions on Medical Imaging, 37(2): 384-395[DOI: 10.1109/TMI.2017.2743464]
Peng C, Lin W A, Liao H F, Chellappa R and Zhou S K. 2020. SAINT: spatially aware interpolation network for medical slice synthesis//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, USA: IEEE: 7747-7756[DOI: 10.1109/CVPR42600.2020.00777http://dx.doi.org/10.1109/CVPR42600.2020.00777]
Retson T A, Besser A H, Sall S, Golden D and Hsiao A. 2019. Machine learning and deep neural networks in thoracic and cardiovascular imaging. Journal of Thoracic Imaging, 34(3): 192-201[DOI: 10.1097/RTI.0000000000000385]
Rispoli V C, Nielsen J F, Nayak K S and Carvalho J L A. 2015. Computational fluid dynamics simulations of blood flow regularized by 3D phase contrast MRI. BioMedical Engineering OnLine, 14(1): #110 [DOI: 10.1186/s12938-015-0104-7]
Sander J, de Vos B D and Išgum I. 2020. Unsupervised super-resolution: creating high-resolution medical images from low-resolution anisotropic examples [EB/OL]. [2021-02-27].https://arxiv.org/pdf/2010.13172.pdfhttps://arxiv.org/pdf/2010.13172.pdf
Shi W Z, Caballero J, Ledig C, Zhuang X H, Bai W J, Bhatia K, de Marvao A M S M, Dawes T, O'Regan D and Rueckert D. 2013. Cardiac image super-resolution with global correspondence using multi-atlas patchmatch//Proceedings of the 16th International Conference on Medical Image Computing and Computer-Assisted Intervention. Nagoya, Japan: Springer: 9-16 [DOI: 10.1007/978-3-642-40760-4_2http://dx.doi.org/10.1007/978-3-642-40760-4_2]
Steeden J A, Quail M, Gotschy A, Mortensen K H, Hauptmann A, Arridge S, Jones R and Muthurangu V. 2020. Rapid whole-heart CMR with single volume super-resolution. Journal of Cardiovascular Magnetic Resonance, 22(1): #56[DOI: 10.1186/s12968-020-00651-x]
Sui Y, Afacan O, Gholipour A and Warfield S K. 2020. Learning a gradient guidance for spatially isotropic MRI super-resolution reconstruction//Proceedings of the 23rd International Conference on Medical Image Computing and Computer-Assisted Intervention. Lima, Peru: Springer: 136-146[DOI: 10.1007/978-3-030-59713-9_14http://dx.doi.org/10.1007/978-3-030-59713-9_14]
Tian Y P, Zhang Y L, Fu Y and Xu C L. 2020. TDAN: temporally-deformable alignment network for video super-resolution//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, USA: IEEE: 3357-3366[DOI: 10.1109/CVPR42600.2020.00342http://dx.doi.org/10.1109/CVPR42600.2020.00342]
Wang X L, Girshick R, Gupta A and He K M. 2018. Non-local neural networks//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA: IEEE: 7794-7803[DOI: 10.1109/CVPR.2018.00813http://dx.doi.org/10.1109/CVPR.2018.00813]
Xiang X Y, Tian Y P, Zhang Y L, Fu Y, Allebach J. P and Xu C L. 2020. Zooming slow-mo: fast and accurate one-stage space-time video super-resolution//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, USA: IEEE: 3367-3376[DOI: 10.1109/CVPR42600.2020.00343http://dx.doi.org/10.1109/CVPR42600.2020.00343]
Zhao C, Shao M H, Carass A, Li H, Dewey B E, Ellingsen L M, Woo J, Guttman M A, Blitz A M, Stone M, Calabresi P A, Halperin H and Prince J L. 2019. Applications of a deep learning method for anti-aliasing and super-resolution in MRI. Magnetic Resonance Imaging, 64: 132-141 [DOI: 10.1016/j.mri.2019.05.038]
Zhao M, Liu X H, Liu H and Wong K K L. 2020. Super-resolution of cardiac magnetic resonance images using laplacian pyramid based on generative adversarial networks. Computerized Medical Imaging and Graphics, 80: #101698 [DOI: 10.1016/j.compmedimag.2020.101698]
Zhou B and Zhou S K. 2020. DuDoRNet: learning a dual-domain recurrent network for fast MRI reconstruction with deep T1 prior//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, USA: IEEE: 4272-4281 [DOI: 10.1109/CVPR42600.2020.00433http://dx.doi.org/10.1109/CVPR42600.2020.00433]
Zhou B, Li C H and Chen W. 2021. Region-level channel attention for single image super-resolution combining high frequency loss. Journal of Image and Graphics, 26(12): 2836-2847
周波, 李成华, 陈伟. 2021. 区域级通道注意力融合高频损失的图像超分辨率重建. 中国图象图形学报, 26(12): 2836-2847[DOI:10. 11834/jig.200582]
Zhu G M, Jiang B, Tong L, Xie Y, Zaharchuk G and Wintermark M. 2019. Applications of deep learning to neuro-imaging techniques. Frontiers in Neurology, 10: #869 [DOI: 10.3389/FNEUR.2019.00869]
相关作者
相关机构
京公网安备11010802024621