浏览全部资源
扫码关注微信
纸质出版日期： 2024-12-16 ，
移动端阅览
胡非易， 张辉， 袁小芳， 刘嘉轩， 陈煜嵘. 2024. 自监督提取光谱序列和语义信息的胆管癌显微高光谱图像分类. 中国图象图形学报， 29(12):3817-3832
Hu Feiyi， Zhang Hui， Yuan Xiaofang， Liu Jiaxuan， Chen Yurong. 2024. Self-supervised extraction of spectral sequence and semantic information for microscopic cholangiocarcinoma hyperspectral image classification. Journal of Image and Graphics， 29(12):3817-3832
目的
2
病理切片的显微高光谱图像包含生物组织反射的光谱信息，为胆管癌组织细胞的分类诊断提供基础。目前病理高光谱图像分类算法的性能大多依赖于高质量标注数据集，然而数据标注过程耗时、费力。基于自监督的特征提取算法可以缓解数据标注难题。因此，本文提出了自监督提取光谱序列和语义信息的胆管癌显微高光谱图像分类方法，提升自监督方法的特征提取能力及分类精度。
方法
2
首先，从自然语言处理中借鉴了具有序列信息建模能力的Transformer架构，将高光谱图像每个像素反射的光谱曲线看做一个光谱序列，设计Transformer自编码器，通过位置嵌入和注意力模块有效关注光谱序列间的差异，从而更好地学习到光谱序列信息。其次，图像经Transformer编码器结构图像增强得到正样本后，设计卷积自编码器作为另一组图像增强，获取对比学习需要的负样本。随后通过新颖的原型对比学习网络捕获图像中的高级语义信息，网络提取特征的过程使用未标记数据。最后，通过少量标记数据微调下游分类任务网络得到分类结果。
结果
2
在多维胆管癌病理高光谱数据集的8个场景上进行实验，结果表明，与现有7种有监督的特征提取方法和5种无监督的特征提取算法相比，本文方法提取的特征在下游分类任务中能达到更高的分类精度，平均总体分类精度达到96.63%。
结论
2
本文方法能从未标记的胆管癌显微高光谱图像中提取有效特征，特征应用于分类任务中达到较高的分类精度，缓解了病理高光谱图像数据标注难题，对胆管癌的医学诊断具有一定的研究价值和现实意义。
Objective
2
Cholangiocarcinoma is a type of cancer with high fatality rate， and the early detection and treatment of cancer can significantly reduce its incidence. Digital diagnosis of pathological sections can effectively improve the accuracy and efficiency of cancer diagnosis. Microscopic hyperspectral images of pathological sections contain richer spectral information than color images. Due to the specific spectral response of biological tissues， pathological tissues have different spectral characteristics from normal tissues， and meaningful and rich spectral information provides great potential for the classification of cancer cells and healthy cells. The performance of most pathologic hyperspectral image classification algorithms is highly dependent on high-quality labeled datasets， but pathologic hyperspectral images need to be manually labeled by experienced pathologists， which can be time consuming and laborious. The feature extraction algorithm based on self-supervision initially extracts features from unlabeled image data in an unsupervised way by designing pretext tasks and then transfers these image data to downstream tasks. After fine-tuning the downstream task network with a limited number of labeled samples， these algorithms can achieve a supervised learning performance and alleviate the data annotation problem. However， traditional contrast self-supervised learning shows limitations in extracting high-level semantic information， and an image enhancement method specific to pathological hyperspectral images is not yet available. Therefore， this paper proposes a self-supervised method to extract sequential spectral data and semantic information from hyperspectral images of cholangiocarcinoma and improve the feature extraction capability and classification accuracy of the self-supervised method.
Method
2
Hyperspectral images are different from natural images in that image enhancement techniques， such as color transformation， can change spectral information. It is meaningful to use the encoder structure as an image enhancement method for hyperspectral images. However， the encoder used in existing methods is based on the convolutional neural network （CNN）， and the feature map extracted by the CNN corresponds to the local receptive field and ignores the global information of the spectral dimension. Given the limited ability of CNN in characterizing spectral sequence data， this paper first designs a Transformer encoder structure for image enhancement， which retains the details of the sequence in the original image. Borrowing from natural language processing， the Transformer architecture with sequential information modeling capability takes the spectral curve reflected by each pixel of the hyperspectral image as a spectral sequence. Transformer then uses position embedding and attention module to pay attention to the differences among spectral sequences and to efficiently learn spectral sequence information. Second， after the image is enhanced with a Transformer encoder structure to obtain positive samples， the convolutional autoencoder can be used as another set of image enhancement to obtain negative samples required for contrastive learning. To address the problem where traditional contrastive learning extracts features through low-level image differences， thus resulting in its limited ability to extract advanced semantic information， this paper applies prototypical contrastive learning to extract features from pathological hyperspectral images. Positive and negative samples are trained through the clustering and instance discrimination tasks of a prototypical contrastive learning network to learn advanced semantic information in images. The above process of extracting features from network structures uses unlabeled data. Finally， the classification results are obtained by fine-tuning the downstream classification task network with a few labeled features.
Result
2
Experiments were conducted on eight scenes in the hyperspectral dataset of multidimensional cholangiocarcinoma pathology. These scenes were selected from eight patients. To ensure the representativeness of these scenes， different cancer cell morphology， cancer cell proportion， and spectral response curve were used in each scene. The proportion of cancer regions in scenes 2， 3， and 8 only accounted for 1/8 of the whole picture. Experiments were conducted on each scenario， where 5% of the data was labeled for training and 95% was used for testing. To verify the effectiveness of the proposed self-supervised method proposed on pathological hyperspectral datasets， this method was compared with 12 widely used algorithms and networks， including 7 supervised feature extraction methods and 5 unsupervised feature extraction algorithms. Experimental results show that the features extracted by the proposed method achieve optimal results in downstream classification tasks， with an average overall accuracy of 96.63%， average accuracy of reaching 95.37%， and average Kappa coefficient of 0.91. Ablation experiments were also conducted to verify that compared with the convolutional module， the Transformer module pays more attention to sequence details when extracting features after adding the self-attention mechanism and multi-head attention mechanism， which can effectively retain original image information and achieve high classification accuracy. The prototypical contrastive learning module adds a clustering process on the basis of contrastive learning and achieves high classification accuracy， thereby proving that the prototypical contrastive learning module can effectively extract high-level semantic information from microscopic hyperspectral images of cholangiocarcinoma. Results of the dimensionality reduction experiment also show that the semantic features extracted by the proposed method are linearly separable.
Conclusion
2
The proposed method can extract effective features from unlabeled hyperspectral images of cholangiocarcinoma， and these features can be applied to classification tasks to achieve high classification accuracy and alleviate the problem of pathological hyperspectral image data labeling. This method carries certain research value and practical significance for the medical diagnosis of cholangiocarcinoma.
癌症分类高光谱图像深度学习自监督学习图像增强
cancer classificationhyperspectral imagesdeep learningself-supervised learningimage enhancement
Bejnordi B E， Veta M， van Diest P J， van Ginneken B， Karssemeijer N， Litjens G， van der Laak J A W M and the CAMELYON16Consortium. 2017. Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. JAMA， 318（22）： 2199-2210 ［DOI： 10.1001/jama.2017.14585http://dx.doi.org/10.1001/jama.2017.14585］
Brindley P J， Bachini M， Ilyas S I， Khan S A， Loukas A， Sirica A E， Teh B T， Wongkham S and Gores G J. 2021. Cholangiocarcinoma. Nature Reviews Disease Primers， 7（1）： #65 ［DOI： 10.1038/s41572-021-00300-2http://dx.doi.org/10.1038/s41572-021-00300-2］
Cao Z Y， Li X R， Feng Y M， Chen S H， Xia C Q and Zhao L Y. 2021. ContrastNet： unsupervised feature learning by autoencoder and prototypical contrastive learning for hyperspectral imagery classification. Neurocomputing， 460： 71-83 ［DOI： 10.1016/j.neucom.2021.07.015http://dx.doi.org/10.1016/j.neucom.2021.07.015］
Caron M， Bojanowski P， Joulin A and Douze M. 2018. Deep clustering for unsupervised learning of visual features//Proceedings of the 15th European Conference on Computer Vision （ECCV）. Munich， Germany： Springer： 139-156 ［DOI： 10.1007/978-3-030-01264-9_9http://dx.doi.org/10.1007/978-3-030-01264-9_9］
Chen T， Kornblith S， Norouzi M and Hinton G E. 2020. A simple framework for contrastive learning of visual representations//Proceedings of the 37th International Conference on Machine Learning. ［s.l.］： PMLR： 1597-1607
Chen X L and He K M. 2021. Exploring simple Siamese representation learning//Proceedings of 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville， USA： IEEE： 15745-15753 ［DOI： 10.1109/CVPR46437.2021.01549http://dx.doi.org/10.1109/CVPR46437.2021.01549］
Chen Y S， Jiang H L， Li C Y， Jia X P and Ghamisi P. 2016. Deep feature extraction and classification of hyperspectral images based on convolutional neural networks. IEEE Transactions on Geoscience and Remote Sensing， 54（10）： 6232-6251 ［DOI： 10.1109/TGRS.2016.2584107http://dx.doi.org/10.1109/TGRS.2016.2584107］
Cinar U， Cetin Atalay R and Cetin Y Y. 2023. Human hepatocellular carcinoma classification from H&E stained histopathology images with 3D convolutional neural networks and focal loss function. Journal of Imaging， 9（2）： #25 ［DOI： 10.3390/jimaging9020025http://dx.doi.org/10.3390/jimaging9020025］
Ding Y， Zhang Z L， Zhao X F， Cai Y M， Li S Y， Deng B and Cai W W. 2022. Self-supervised locality preserving low-pass graph convolutional embedding for large-scale hyperspectral image clustering. IEEE Transactions on Geoscience and Remote Sensing， 60： #5536016 ［DOI： 10.1109/TGRS.2022.3198842http://dx.doi.org/10.1109/TGRS.2022.3198842］
Gao H M， Zhu M， Cao X Y， Li C M， Liu Q and Xu P P. 2023. A micro-hyperspectral image classification method of gallbladder cancer based on multi-scale fusion attention mechanism. Journal of Image and Graphics， 28（4）： 1173-1185
高红民， 朱敏， 曹雪莹， 李臣明， 刘芹， 许佩佩. 2023. 多尺度融合注意力机制的胆囊癌显微高光谱图像分类. 中国图象图形学报， 28（4）： 1173-1185 ［DOI： 10.11834/jig.211201http://dx.doi.org/10.11834/jig.211201］
Grill J B， Strub F， Altché F， Tallec C， Richemond P H， Buchatskaya E， Doersch C， Pires B A， Guo Z D， Azar M G， Piot B， Kavukcuoglu K， Munos R and Valko M. 2020. Bootstrap your own latent a new approach to self-supervised learning//Proceedings of the 34th International Conference on Neural Information Processing Systems. Vancouver， Canada： Curran Associates Inc.： #1786
He K M， Fan H Q， Wu Y X， Xie S N and Girshick R. 2020. Momentum contrast for unsupervised visual representation learning//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle， USA： IEEE： 9726-9735 ［DOI： 10.1109/CVPR42600.2020.00975http://dx.doi.org/10.1109/CVPR42600.2020.00975］
Hong D F， Han Z， Yao J， Gao L R， Zhang B， Plaza A and Chanussot J. 2022. SpectralFormer： rethinking hyperspectral image classification with Transformers. IEEE Transactions on Geoscience and Remote Sensing， 60： #5518615 ［DOI： 10.1109/TGRS.2021.3130716http://dx.doi.org/10.1109/TGRS.2021.3130716］
Hou S K， Shi H Y， Cao X H， Zhang X H and Jiao L C. 2021. Hyperspectral imagery classification based on contrastive learning. IEEE Transactions on Geoscience and Remote Sensing， 60： #5521213 ［DOI： 10.1109/TGRS.2021.3139099http://dx.doi.org/10.1109/TGRS.2021.3139099］
Hu X， Li T， Zhou T， Liu Y and Peng Y X. 2021. Contrastive learning based on Transformer for hyperspectral image classification. Applied Sciences， 11（18）： #8670 ［DOI： 10.3390/app11188670http://dx.doi.org/10.3390/app11188670］
Huang J B， Gong S G and Zhu X T. 2020. Deep semantic clustering by partition confidence maximisation//Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition （CVPR）. Seattle， USA： IEEE： 8846-8855 ［DOI： 10.1109/CVPR42600.2020.00887http://dx.doi.org/10.1109/CVPR42600.2020.00887］
Huang L B， Chen Y S and He X. 2023. Spectral-spatial masked Transformer with supervised and contrastive learning for hyperspectral image classification. IEEE Transactions on Geoscience and Remote Sensing， 61： #5508718 ［DOI： 10.1109/TGRS.2023.3264235http://dx.doi.org/10.1109/TGRS.2023.3264235］
Huang X， Dong M J， Li J Y and Guo X. 2022. A 3-D-swin Transformer-based hierarchical contrastive learning method for hyperspectral image classification. IEEE Transactions on Geoscience and Remote Sensing， 60： #5411415 ［DOI： 10.1109/TGRS.2022.3202036http://dx.doi.org/10.1109/TGRS.2022.3202036］
Ilyas S I， Khan S A， Hallemeier C L， Kelley R K and Gores G J. 2018. Cholangiocarcinoma—evolving concepts and therapeutic strategies. Nature Reviews Clinical Oncology， 15（2）： 95-111 ［DOI： 10.1038/nrclinonc.2017.157http://dx.doi.org/10.1038/nrclinonc.2017.157］
Kingma D P and Welling M. 2014. Auto-encoding variational Bayes//Proceedings of the 2nd International Conference on Learning Representations. Banff， Canada： ［s.n.］
Koohbanani N A， Unnikrishnan B， Khurram S A， Krishnaswamy P and Rajpoot N. 2021. Self-path： self-supervision for classification of pathology images with limited annotations. IEEE Transactions on Medical Imaging， 40（10）： 2845-2856 ［DOI： 10.1109/TMI.2021.3056023http://dx.doi.org/10.1109/TMI.2021.3056023］
Li J， Chen J Y， Tang Y C， Wang C， Landman B A and Zhou S K. 2023. Transforming medical imaging with Transformers？ A comparative review of key properties， current progresses， and future perspectives. Medical Image Analysis， 85： #102762 ［DOI： 10.1016/j.media.2023.102762http://dx.doi.org/10.1016/j.media.2023.102762］
Li J N， Zhou P， Xiong C M and Hoi S C H. 2021. Prototypical contrastive learning of unsupervised representations//Proceedings of the 9th International Conference on Learning Representations. Austria： OpenReview.net ［DOI： 10.48550/arXiv.2005.04966http://dx.doi.org/10.48550/arXiv.2005.04966］
Li S T， Song W W， Fang L Y， Chen Y S， Ghamisi P and Benediktsson J A. 2019. Deep learning for hyperspectral image classification： an overview. IEEE Transactions on Geoscience and Remote Sensing， 57（9）： 6690-6709 ［DOI： 10.1109/TGRS.2019.2907932http://dx.doi.org/10.1109/TGRS.2019.2907932］
Li Z T. 2020. Research on Land Cover Classification Algorithms with Spectral Data Based on Attention Mechanism. Wuhan： Huazhong University of Science and Technology （李正涛. 2020. 基于注意力机制的光谱地物分类方法研究. 武汉： 华中科技大学） ［DOI： 10.27157/d.cnki.ghzku.2020.001870］
Litjens G， Sánchez C I， Timofeeva N， Hermsen M， Nagtegaal I， Kovacs I， Hulsbergen - van de Kaa C， Bult P， van Ginneken B and van der Laak J. 2016. Deep learning as a tool for increased accuracy and efficiency of histopathological diagnosis. Scientific Reports， 6： #26286 ［DOI： 10.1038/srep26286http://dx.doi.org/10.1038/srep26286］
Liu B， Yu A Z， Yu X C， Wang R R， Gao K L and Guo W Y. 2021. Deep multiview learning for hyperspectral image classification. IEEE Transactions on Geoscience and Remote Sensing， 59（9）： 7758-7772 ［DOI： 10.1109/TGRS.2020.3034133http://dx.doi.org/10.1109/TGRS.2020.3034133］
Liu S S， Tan Y Z and Lu P. 2022. New progress in basic and clinical research of photodynamic therapy for cholangiocarcinoma. Journal of Clinical Surgery， 30（11）： 1094-1097
刘淞淞， 谭一舟， 卢鹏. 2022. 胆管癌光动力治疗的基础与临床研究新进展. 临床外科杂志， 30（11）： 1094-1097 ［DOI： 10.3969/j.issn.1005-6483.2022.11.028http://dx.doi.org/10.3969/j.issn.1005-6483.2022.11.028］
Lu G L and Fei B W. 2014. Medical hyperspectral imaging： a review. Journal of Biomedical Optics， 19（1）： #010901 ［DOI： 10.1117/1.JBO.19.1.010901http://dx.doi.org/10.1117/1.JBO.19.1.010901］
Makantasis K， Karantzalos K， Doulamis A and Doulamis N. 2015. Deep supervised learning for hyperspectral data classification through convolutional neural networks//2015 IEEE International Geoscience and Remote Sensing Symposium （IGARSS）. Milan， Italy： IEEE： 4959-4962 ［DOI： 10.1109/IGARSS.2015.7326945http://dx.doi.org/10.1109/IGARSS.2015.7326945］
Makhzani A， Shlens J， Jaitly N， Goodfellow I and Frey B. 2015. Adversarial autoencoders ［EB/OL］. ［2023-12-11］. https://doi.org/10.48550/arXiv.1511.05644.pdfhttps://doi.org/10.48550/arXiv.1511.05644.pdf
Noble W S. 2006. What is a support vector machine？Nature Biotechnology， 24（12）： 1565-1567 ［DOI： 10.1038/nbt1206-1565http://dx.doi.org/10.1038/nbt1206-1565］
Qing Y， Liu W Y， Feng L Y and Gao W J. 2021. Improved Transformer net for hyperspectral image classification. Remote Sensing， 13（11）： #2216 ［DOI： 10.3390/rs13112216http://dx.doi.org/10.3390/rs13112216］
Ronneberger O， Fischer P and Brox T. 2015. U-Net： convolutional networks for biomedical image segmentation//Proceedings of the 18th International Conference on Medical Image Computing and Computer-Assisted Intervention. Munich， Germany： Springer： 234-241 ［DOI： 10.1007/978-3-319-24574-4_28http://dx.doi.org/10.1007/978-3-319-24574-4_28］
Roy S K， Krishna G， Dubey S R and Chaudhuri B B. 2020. HybridSN： exploring 3-D-2-D CNN feature hierarchy for hyperspectral image classification. IEEE Geoscience and Remote Sensing Letters， 17（2）： 277-281 ［DOI： 10.1109/LGRS.2019.2918719http://dx.doi.org/10.1109/LGRS.2019.2918719］
Sheng Z T， Chen Y X and Qi G J. 2023. Audio-visual adversarial contrastive learning-based multi-modal self-supervised feature fusion. Journal of Image and Graphics， 28（1）： 317-332
盛振涛， 陈雁翔， 齐国君. 2023. 面向多模态自监督特征融合的音视频对抗对比学习. 中国图象图形学报， 28（1）： 317-332 ［DOI： 10.11834/jig.220168http://dx.doi.org/10.11834/jig.220168］
Sun H， Zheng X T， Lu X Q and Wu S Y. 2020. Spectral-spatial attention network for hyperspectral image classification. IEEE Transactions on Geoscience and Remote Sensing， 58（5）： 3232-3245 ［DOI： 10.1109/TGRS.2019.2951160http://dx.doi.org/10.1109/TGRS.2019.2951160］
Sun L， Zhao G R， Zheng Y H and Wu Z B. 2022. Spectral-spatial feature tokenization Transformer for hyperspectral image classification. IEEE Transactions on Geoscience and Remote Sensing， 60： #5522214 ［DOI： 10.1109/TGRS.2022.3144158http://dx.doi.org/10.1109/TGRS.2022.3144158］
Trajanovski S， Shan C F， Weijtmans P J C， de Koning S G B and Ruers T J M. 2021. Tongue tumor detection in hyperspectral images using deep learning semantic segmentation. IEEE Transactions on Biomedical Engineering， 68（4）： 1330-1340 ［DOI： 10.1109/TBME.2020.3026683http://dx.doi.org/10.1109/TBME.2020.3026683］
Wang Q， Sun L， Wang Y， Zhou M， Hu M H， Chen J G， Wen Y and Li Q L. 2021. Identification of melanoma from hyperspectral pathology image using 3D convolutional networks. IEEE Transactions on Medical Imaging， 40（1）： 218-227 ［DOI： 10.1109/TMI.2020.3024923http://dx.doi.org/10.1109/TMI.2020.3024923］
Wang Q， Wang J B， Zhou M， Li Q L and Wang Y T. 2017. Spectral-spatial feature-based neural network method for acute lymphoblastic leukemia cell identification via microscopic hyperspectral imaging technology. Biomedical Optics Express， 8（6）： 3017-3028 ［DOI： 10.1364/BOE.8.003017http://dx.doi.org/10.1364/BOE.8.003017］
Xu Y H， Du B and Zhang L P. 2024. Robust self-ensembling network for hyperspectral image classification. IEEE Transactions on Neural Networks and Learning Systems， 35（3）： 3780-3793 ［DOI： 10.1109/TNNLS.2022.3198142http://dx.doi.org/10.1109/TNNLS.2022.3198142］
Yu K H， Zhang C， Berry G J， Altman R B， Ré C， Rubin D L and Snyder M. 2016. Predicting non-small cell lung cancer prognosis by fully automated microscopic pathology image features. Nature Communications， 7： #12474 ［DOI： 10.1038/ncomms12474http://dx.doi.org/10.1038/ncomms12474］
Yu W B， Zhang M and Shen Y. 2021. Spatial revising variational autoencoder-based feature extraction method for hyperspectral images. IEEE Transactions on Geoscience and Remote Sensing， 59（2）： 1410-1423 ［DOI： 10.1109/TGRS.2020.2997835http://dx.doi.org/10.1109/TGRS.2020.2997835］
Zhang Q， Li Q L， Yu G Z， Sun L， Zhou M and Chu J H. 2019. A multidimensional choledoch database and benchmarks for cholangiocarcinoma diagnosis. IEEE Access， 7： 149414-149421 ［DOI： 10.1109/ACCESS.2019.2947470http://dx.doi.org/10.1109/ACCESS.2019.2947470］
Zhang R， Yang Y X， Li Y， Wang J B， Miao Z， Li H and Wang Z Q. 2022. Self-supervised learning based few-shot remote sensing scene image classification. Journal of Image and Graphics， 27（11）： 3371-3381
张睿， 杨义鑫， 李阳， 王家宝， 苗壮， 李航， 王梓祺. 2022. 自监督学习下小样本遥感图像场景分类. 中国图象图形学报， 27（11）： 3371-3381 ［DOI： 10.11834/jig.210486http://dx.doi.org/10.11834/jig.210486］
Zhang Y， Wang Y， Zhang B Y and Li Q L. 2022. A hyperspectral dataset of precancerous lesions in gastric cancer and benchmarks for pathological diagnosis. Journal of Biophotonics， 15（11）： #e202200163 ［DOI： 10.1002/jbio.202200163http://dx.doi.org/10.1002/jbio.202200163］
Zhao F， Zhang J J， Meng Z， Liu H Q， Chang Z H and Fan J L. 2023. Multiple vision architectures-based hybrid network for hyperspectral image classification. Expert Systems with Applications， 234： #121032 ［DOI： 10.1016/j.eswa.2023.121032http://dx.doi.org/10.1016/j.eswa.2023.121032］
Zheng S J， Qiu S， Li Q L， Zhou M， Hu M H and Yu G Z. 2021. Fourier transform channel attention network for cholangiocarcinoma hyperspectral image segmentation. Journal of Image and Graphics， 26（8）： 1836-1846
郑少佳， 邱崧， 李庆利， 周梅， 胡孟晗， 于观贞. 2021. 傅里叶变换通道注意力网络的胆管癌高光谱图像分割. 中国图象图形学报， 26（8）： 1836-1846 ［DOI： 10.11834/jig.210207http://dx.doi.org/10.11834/jig.210207］
相关作者
相关机构
京公网安备11010802024621