浏览全部资源
扫码关注微信
纸质出版日期： 2021-09-16 ，
录用日期： 2021-02-25
移动端阅览
王黎, 曹颖, 田梨梨, 陈祈剑, 郭顺超, 张健, 王丽会. 自适应多模态特征融合胶质瘤分级网络[J]. 中国图象图形学报, 2021,26(9):2243-2256.
Li Wang, Ying Cao, Lili Tian, Qijian Chen, Shunchao Guo, Jian Zhang, Lihui Wang. Adaptive multi-modality fusion network for glioma grading[J]. Journal of Image and Graphics, 2021,26(9):2243-2256.
目的
2
胶质瘤的准确分级是辅助制定个性化治疗方案的主要手段，但现有研究大多数集中在基于肿瘤区域的分级预测上，需要事先勾画感兴趣区域，无法满足临床智能辅助诊断的实时性需求。因此，本文提出一种自适应多模态特征融合网络（adaptive multi-modal fusion net，AMMFNet），在不需要勾画肿瘤区域的情况下，实现原始采集图像到胶质瘤级别的端到端准确预测。
方法
2
AMMFNet方法采用4个同构异义网络分支提取不同模态的多尺度图像特征；利用自适应多模态特征融合模块和降维模块进行特征融合；结合交叉熵分类损失和特征嵌入损失提高胶质瘤的分类精度。为了验证模型性能，本文采用MICCAI（Medical Image Computing and Computer Assisted Intervention Society）2018公开数据集进行训练和测试，与前沿深度学习模型和最新的胶质瘤分类模型进行对比，并采用精度以及受试者曲线下面积（area under curve，AUC）等指标进行定量分析。
结果
2
在无需勾画肿瘤区域的情况下，本文模型预测胶质瘤分级的AUC为0.965；在使用肿瘤区域时，其AUC高达0.997，精度为0.982，比目前最好的胶质瘤分类模型——多任务卷积神经网络同比提高1.2%。
结论
2
本文提出的自适应多模态特征融合网络，通过结合多模态、多语义级别特征，可以在未勾画肿瘤区域的前提下，准确地实现胶质瘤分级预测。
Objective
2
Glioma grading has been a vital research tool for customized treatment in of the glioma. Glioma grading can be an assessment tool for biopsy and histopathological to resolve invasive and time-consuming issues. A non-invasive scheme for grading gliomas precision has played the key role. A reliable non-invasive grading scheme has been implemented for magnetic resonance imaging (MRI)to facilitate computer-assisted diagnosis system(CAD) for glioma grading. The medical image-based grading has been used manual features to implement image-level tumor analysis. Manual feature-based methods have realized higher area under curve (AUC) based on the variation of image intensity and image deformation analyses constrained generalization capability. The emerging deep learning method has projected deep features more semantic and representative compared with the manual features in generalization. The original data has been projected into semantic space to task-level features for data segmentation compared with image-level features. Deep feature-based models have been more qualifier than manual feature-based models on the aspect of classification tasks. Time-consuming and labor-intensive segmentation has been analyzed in lesion regions for tumor. The prediction have been constrained by the tumor segmentation accuracy. A multi-modality fusion network (AMMFNet) has been applied in grading gliomas instead of tumor segmentation.
Method
2
AMMFNet has been an end-to-end multi-scale model to improve glioma grading performance via deep learning-based multi-modal fusion features. The network has contained three components including multi-modal image feature extraction module
adaptive multi-modal and multi-scale features fusion module and classification module respectively. The feature extraction module has been used for deep features extraction based on the images acquired with four different modalities. The width and depth of model has good quality in extracting semantic features. The adaptive multi-modal and multi-scale features fusing module have intended to learn fusion rules via multi-modal and multi-scale deep features. The fusion of the multi-modal features in the same semantic level via high dimensional convolution layers. An adaptive dimension reduction module have adopted to fuse the features in different semantic levels to realize the different shape of feature maps into the same size. Such reduction module has been built up in three-branch structure based on the unique dimension reducing implementation for each branch. Task-level loss and feature-level loss have been used to train the proposed model to upgrade glioma grading accuracy. The task-level loss has been achieved via weighted cross entropy. The feature-level loss has been used to maximize the intra-class feature similarity and inter-class feature discrepancy via the cosine function of two feature vectors. Based on the public Medical Image Computing and Computer Assisted Intervention Society(MICCAI) 2018 challenge dataset to train and test the proposed model
the accuracy(ACC)
specificity(SPE)
sensitivity(SEN)
positive precision value(PPV)
negative precision value(NPV)
average F1 score and AUC have been prior to analyze the grading validation.
Result
2
The AUC has been the highest one with a value of 0.965 comparing with visual geometry group 19-layer net(VGG19)
ResNet
SENet
SEResNet
InceptionV4 and NASNet. ACC
SPE
PPV
F1-score have been increased by 2.8%
2.1%
1.1%
and 3.1% each at most. The tumor region of interest(ROI) modeling input has been trained. The ACC has been increased by 1.2%. Ablation experiments including replacing the deeper convolutional layer with Resblock as well as adding SE block into fusion module have been further validated via customized learning modules. The ACC
SEN
SPE
PPV
NPV
F1 of AMMFNet using SE fusion block have been increased by 0.9%
0.1%
2.5%
1.0%
0.6%
1.2% respectively in the context of baseline.
Conclusion
2
The adaptive multi-modal fusion network has been demonstrated via fused multi-modal features and the multi-scale integration of fused deep features. The multi-modal and multi-scale features integration has been illustrated to capture more expressive features related to image details. The demonstration can be prior to locate the tumor even without lesion areas or segmented tumor. An end-to-end model has the good priority in glioma grading.
胶质瘤分级深度学习多模态融合多尺度特征端到端分类
glioma gradingdeep learningmultimodal feature fusionmultiscale deep featureend-to-end classification model
Aerts H J W L, Velazquez E R, Leijenaar R T H, Parmar C, Grossmann P, Carvalho S, Bussink J, Monshouwer R, Haibe-Kains B, Rietveld D, Hoebers F, Rietbergen M M, Leemans C R, Dekker A, Quackenbush J, Gillies R J and Lambin P. 2014. Decoding tumour phenotype by noninvasive imaging using a quantitative radiomics approach. Nature Communications, 5(1): #4006[DOI: 10.1038/ncomms5006]
Afshar P, Mohammadi A, Plataniotis K N, Oikonomou A and Benali H. 2019. From handcrafted to deep-learning-based cancer radiomics: challenges and opportunities. IEEE Signal Processing Magazine, 36(4): 132-160[DOI: 10.1109/MSP.2019.2900993]
Ali M B, Gu I Y H and Jakola A S. 2019. Multi-stream convolutional autoencoder and 2D generative adversarial network for glioma classification//Proceedings of 2019 International Conference on Computer Analysis of Images and Patterns. Salerno, Italy: Springer: 234-245[DOI: 10.1007/978-3-030-29888-3_19http://dx.doi.org/10.1007/978-3-030-29888-3_19]
Bakas S, Akbari H, Sotiras A, Bilello M, Rozycki M, Kirby J S, Freymann J B, Farahani K and Davatzikos C. 2017. Advancing the cancer genome atlas glioma MRI collections with expert segmentation labels and radiomic features. Scientific Data, 4(1): #170117[DOI: 10.1038/sdata.2017.117]
Cheng D C, Meng G F, ChengG L and Pan C H. 2017. SeNet: structured edge network for sea-land segmentation. IEEE Geoscience and Remote Sensing Letters, 14(2): 247-251[DOI: 10.1109/LGRS.2016.2637439]
Clark K, Vendt B, Smith K, Freymann J, Kirby J, Koppel P, Moore S, Phillips S, Maffitt D, Pringle M, Tarbox L and Prior F. 2013. The cancer imaging archive (TCIA): maintainingand operating a public information repository. Journal of Digital Imaging, 26(6): 1045-1057[DOI: 10.1007/s10278-013-9622-7]
Ge C J, Gu I Y H, Jakola A S and Yang J. 2018a. Deep learning and multi-sensor fusion for glioma classification using multistream 2D convolutional networks//Proceedings of the 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). Honolulu, USA: IEEE: 5894-5897[DOI: 10.1109/EMBC.2018.8513556http://dx.doi.org/10.1109/EMBC.2018.8513556]
Ge C J, Qu Q X, Gu I Y H and Jakola A S. 2018b. 3D multi-scale convolutional networks for glioma grading using MR images//Proceedings of the 25th IEEE International Conference on Image Processing (ICIP). Athens, Greece: IEEE: 141-145[DOI: 10.1109/ICIP.2018.8451682http://dx.doi.org/10.1109/ICIP.2018.8451682]
Gillies R J, Kinahan P E and Hricak H. 2016. Radiomics: images are more than pictures, they are data. Radiology, 278(2): 563-577. [DOI: 10.1148/radiol.2015151169]
He K M, Zhang X Y, Ren S Q and Sun J. 2016. Deep residual learning for image recognition//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vagas, USA: IEEE: 770-778[DOI: 10.1109/CVPR.2016.90http://dx.doi.org/10.1109/CVPR.2016.90]
Hsieh K L C, Chen C Y and Lo C M. 2017. Quantitative glioma grading using transformed gray-scale invariant textures of MRI. Computers in Biology and Medicine, 83: 102-108[DOI: 10.1016/j.compbiomed.2017.02.012]
Jackson R J, Fuller G N, Abi-Said D, Lang F F, Gokaslan Z L, Shi W M, Wildrick D M and Sawaya R. 2001. Limitations of stereotactic biopsy in the initial management of gliomas. Neuro-oncology, 3(3): 193-200[DOI: 10.1093/neuonc/3.3.193]
Jacobsen J H, Van Gemert J, Lou Z Y and Smeulders A W M. 2016. Structured receptive fields in CNNs//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vagas, USA: IEEE: 2610-2619[DOI: 10.1109/CVPR.2016.286http://dx.doi.org/10.1109/CVPR.2016.286]
Lambin P, Leijenaar R T H, Deist T M, Peerlings J, De Jong E E C, Van Timmeren J, Sanduleanu S, Larue R T H M, Even A J G, Jochems A, Van Wijk Y, Woodruff H, Van Soest J, Lustberg T, Roelofs E, Van Elmpt W, Dekker A, Mottaghy F M, Wildberger J E and Walsh S. 2017. Radiomics: the bridge between medical imaging and personalized medicine. Nature Reviews Clinical Oncology, 14(12): 749-762[DOI: 10.1038/nrclinonc.2017.141]
Lambin P, Rios-Velazquez E, Leijenaar R, Carvalho S, Van Stiphout R G P M, Granton P, Zegers C M L, Gillies R, Boellard R, Dekker A and Aerts H J W L. 2012. Radiomics: extracting more information from medical images using advanced feature analysis. European Journal of Cancer, 48(4): 441-446[DOI: 10.1016/j.ejca.2011.11.036]
Li C P and Dong J N. 2020. Advances in preoperative identification of subtypes and lymph node metastasis of cervical cancer by IVIM and texture analysis. Journal of Image and Graphics, 25(10): 2093-2099.
李翠平, 董江宁. 2020. IVIM及纹理分析在术前预测宫颈癌类型和淋巴结转移研究进展. 中国图象图形学报, 25(10): 2093-2099[DOI: 10.11834/jig.200128]
Liu Y P, Liu G P, Wang R F, Jin R, Sun D C, Qiu H, Dong C, Li J and Hong G B. 2020. Accurate segmentation method of liver tumor CT based on the combination of deep learning and radiomics. Journal of Image and Graphics, 25(10): 2128-2141
刘云鹏, 刘光品, 王仁芳, 金冉, 孙德超, 邱虹, 董晨, 李瑾, 洪国斌. 2020. 深度学习结合影像组学的肝脏肿瘤CT分割. 中国图象图形学报, 25(10), 2128-2141[DOI: 10.11834/jig.200198]
Mamelak A N and Jacoby D B. 2007. Targeted delivery of antitumoral therapy to glioma and other malignancies with synthetic chlorotoxin (TM-601). Expert Opinion on Drug Delivery, 4(2): 175-186[DOI: 10.1517/17425247.4.2.175]
Menze B H, Jakab A, Bauer S, Kalpathy-Cramer J, Farahani K, Kirby J, Burren Y, Porz N, Slotboom J, Wiest R, Lanczi L, Gerstner E, Weber M A, Arbel T, Avants B B, Ayache N, Buendia P, Collins D L, Cordier N, Corso J J, Criminisi A, Das T, Delingette H, Demiralp Ç, Durst C R, Dojat M, Doyle S, Festa J, Forbes F, Geremia E, Glocker B, Golland P, Guo X T, Hamamci A, Ifteharuddin K M, Jena R, John N M, Konukoglu E, Lashkari D, Mariz J A, Meier R, Pereira S, Precup D, Price S J, Raviv T R, Reza S M S, Ryan M, Sarikaya D, Schwartz L, Shin H C, Shotton J, Silva C A, Sousa N, Subbana N K, Szekely G, Taylor T J, Thomas O M, Tustison N J, Unal G, Vasseur F, Wintermark M, Ye D H, Zhao L, Zhao B S, Zikic D, Prastawa M, Reyes M and Van Leemput K. 2015. The multimodal brain tumor image segmentation benchmark (BRATS). IEEE Transactions on Medical Imaging, 34(10): 1993-2024[DOI: 10.1109/TMI.2014.2377694]
Mittler M A, Walters B C and Stopa E G. 1996. Observer reliability in histological grading of astrocytoma stereotactic biopsies. Journal of Neurosurgery, 85(6): 1091-1094[DOI: 10.3171/jns.1996.85.6.1091]
Mzoughi H, Njeh I, Wali A, Slima M B, BenHamida A, Mhiri C and Mahfoudhe K B. 2020. Deep multi-scale 3D convolutional neural network (CNN) for MRI gliomas brain tumor classification. Journal of Digital Imaging, 33(4): 903-915[DOI: 10.1007/s10278-020-00347-9]
Qin J B, Liu Z Y, Zhang H, Shen C, Wang X C, Tan Y, Wang S, Wu X F and Tian J. 2017. Grading of gliomas by using radiomic features on multiple magnetic resonance imaging (MRI) sequences. Medical Science Monitor, 23: 2168-2178[DOI: 10.12659/MSM.901270]
Simonyan K and Zisserman A. 2015. Very deep convolutional networks for large-scale image recognition[EB/OL]. [2020-02-07].https://arxiv.org/pdf/1409.1556.pdfhttps://arxiv.org/pdf/1409.1556.pdf
Sun M L, Song Z J, Jiang X H, Pan J and Pang Y W. 2017. Learning pooling for convolutional neural network. Neurocomputing, 224: 96-104[DOI: 10.1016/j.neucom.2016.10.049]
Szegedy C, Ioffe S, Vanhoucke V and Alemi A. 2016. Inception-v4, inception-resnet and the impact of residual connections on learning[EB/OL]. [2020-02-07].https://arxiv.org/pdf/1602.07261.pdfhttps://arxiv.org/pdf/1602.07261.pdf
Targ S, Almeida D and Lyman K. 2016. Resnet in resnet: generalizing residual architectures[EB/OL]. [2020-12-11].https://arxiv.org/pdf/1603.08029.pdfhttps://arxiv.org/pdf/1603.08029.pdf.
Tian Q, Yan L F, Zhang X, Zhang X, Hu Y C, Han Y, Liu Z C, Nan H Y, Sun Q, Sun Y Z, Yang Y, Yu Y, Zhang J, Hu B, Xiao G, Chen P, Tian S, Xu J, Wang W and Cui G B. 2018. Radiomics strategy for glioma grading using texture features from multiparametric MRI. Journal of Magnetic Resonance Imaging, 48(6): 1518-1528[DOI: 10.1002/jmri.26010]
Vamvakas A, Williams S C, Theodorou K, Kapsalaki E, Fountas K, Kappas C, Vassiou K and Tsougos I. 2019. Imaging biomarker analysis of advanced multiparametric MRI for glioma grading. Physica Medica, 60: 188-198[DOI: 10.1016/j.ejmp.2019.03.014]
Wen P Y and Reardon D A. 2016. Progress in glioma diagnosis, classification and treatment. Nature Reviews Neurology, 12(2): 69-71[DOI: 10.1038/nrneurol.2015.242]
Wesseling P and Capper D. 2018. WHO 2016 classification of gliomas. Neuropathology and Applied Neurobiology, 44(2): 139-150[DOI: 10.1111/nan.12432]
Yang Y, Yan L F, Zhang X, Han Y, Nan H Y, Hu Y C, Hu B, Yan S L, Zhang J, Cheng D L, Ge X W, Cui G B, Zhao D and Wang W. 2018. Glioma grading on conventional MR images: a deep learning study with transfer learning. Frontiers in Neuroscience, 12: #804[DOI: 10.3389/fnins.2018.00804]
Yip S S F and Aerts H J W L. 2016. Applications and limitations of radiomics. Physics in Medicine and Biology, 61(13): #R150[DOI: 10.1088/0031-9155/61/13/R150]
Zhuge Y, Ning H, Mathen P, Cheng J Y, Krauze A V, Camphausen K and Miller R W. 2020. Automated glioma grading on conventional MRI images using deep convolutional neural networks. Medical Physics, 47(7): 3044-3053[DOI: doi.org/10.1002/mp.14168]
Zoph B, Vasudevan V, Shlens J and Le Q V. 2018. Learning transferable architectures for scalable image recognition//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA: IEEE: 8697-8710[DOI: 10.1109/CVPR.2018.00907http://dx.doi.org/10.1109/CVPR.2018.00907]
相关作者
相关机构
京公网安备11010802024621