Objective

2

Medical report generation leverages natural language processing and machine learning techniques to convert medical data， such as medical images， into structured text reports. This aims to improve efficiency in the medical field and reduce the worklord of healthcare professionals. Annually， over 3.6 billion X-ray imaging examinations are conducted worldwide， with the majority being chest X-ray， and this number continues rise. This trend places increasing pressure on radiologists， impacting both the quality and speed of clinical decision-making. Inspired by various advanced general image captioning methods， current medical report generation methods can be broadly categorized into three types based on their implementation techniques： encoder-decoder methods， cross-modal alignment methods， and knowledge-enhanced methods. Among these， encoder-decoder methods， particularly those utilizing the Transformer architecture， are the most widely adopted. The Transformer architecture excels at encoding long-term dependencies and learning efficient feature representations， making it very suitable for medical report generation tasks. However， most of these methods employ an end-to-end approach than generates only diagnostic reports without classifying the types of chest diseases， thus lacking the assistance of disease label semantic information. Despite significant progress in medical report generation research driven by advances in deep learning， several challenges remain. First， there exists a semantic gap between medical images and text reports. Most existing methods focus only on the aligning prominent information in the images and text， often neglecting the interaction of fine-grained information， which hampers the association between local information in detailed image regions and text reports. Second， the complexity and diversity of organs and diseases in chest X-ray images， along with special cases such as complications and disease coexistence， make it difficult to generate accurate text reports. Additionally， key clinical information in medical report generation often derives from descriptions of abnormalities， and the absence of abnormalities in both the images and reports frequently leads models to generate normal and similar reports. To address these issues， this paper proposes a chest X-ray image report generation method that combines knowledge enhancement and feature alignment. The approach includes three main modules： first， a feature representation module extracts more detailed features from text reports and chest images ； second， Knowledge-enhanced visual features incorporate prior medical knowledge， guiding the learning of visual features； and third， global-local feature alignment promotes semantic alignment between images， reports， and disease labels， thereby improving the accuracy and completeness of the generated reports.

Method

2

First， chest X-ray images and text reports are input， and an image and text feature representation module is constructed， which includes visual and textual encoders to extract global and local features from the images and text， respectively. Then， a chest prior knowledge graph is introduced， enabling knowledge-enhanced visual feature learning through the encoding of pathological image knowledge， resulting in enhanced visual features after fusion. Finally， cross-attention is defined to align the global-local features of the image and text with the visual-disease labels across modalities， and multi-head attention is used in the encoder-decoder to generate accurate chest X-ray image reports.

Results

2

The effectiveness of the proposed method was validated through comparative experiments on two challenging datasets， IU X-Ray and MIMIC-CXR. The results show that in the IU X-Ray dataset， the BLEU-1， 3， and 4 scores reached 0.505， 0.235， and 0.178， respectively， representing an improvement over most existing methods for the same task. This indicates that the proposed model offers certain advantages in text fluency and better focus on disease regions， generating correct label information， significantly improving various report metrics. The CIDEr and ROUGE-L scores are also comparable to those of other methods， demonstrating that knowledge enhancement and feature alignment positively impact the quality of the generated reports. On the MIMIC-CXR dataset， the BLEU-2 and 3 metrics increased by 0.4% and 1.2%， respectively， proving the robustness of the model， which maintains its advantages even when faced with more complex and diverse data. The CE metrics in the MIMIC-CXR dataset also showed improvement， with the precision metric reaching 0.428， recall reaching 0.343， and F1 reaching 0.360， indicating the model's effectiveness in generating complete and consistent reports. Qualitative experiments show that the proposed method generates medical reports largely consistent with reference reports. Additionally， for abnormal images， the model accurately locates the abnormal regions， with the generated reports including not only descriptions of the disease conditions but also describe the lesions. The vocabulary used is professional and closely aligns with the ground truth reports， demonstrating that the knowledge enhancement contributes to the model's ability to generate professional and accurate text. Ablation experiments confirm that the inclusion of image and text feature representation， knowledge enhancement， and feature alignment modules improves report generation compared to the basic model. This suggests that incorporating external knowledge to the medical report generation model and fully utilizing the visual features of similar medical images enables the model learn sufficient visual features of medical images during the encoding stage， guiding the model to generate more complete and accurate medical reports during the decoding stage.

Conclusion

2

The chest X-ray image report generation method proposed in this paper captures detailed features of both images and text， focus on the relationships between global-local features and disease categories， and enhance the alignment between images and text， enabling the generation of complete and accurate medical reports.<p>Objective Medical report generation leverages natural language processing and machine learning techniques to convert medical data， such as medical images， into structured text reports. This aims to improve efficiency in the medical field and reduce the worklord of healthcare professionals. Annually， over 3.6 billion X-ray imaging examinations are conducted worldwide， with the majority being chest X-ray， and this number continues rise. This trend places increasing pressure on radiologists， impacting both the quality and speed of clinical decision-making. Inspired by various advanced general image captioning methods， current medical report generation methods can be broadly categorized into three types based on their implementation techniques： encoder-decoder methods， cross-modal alignment methods， and knowledge-enhanced methods. Among these， encoder-decoder methods， particularly those utilizing the Transformer architecture， are the most widely adopted. This paper proposes a chest X-ray image report generation method that combines knowledge enhancement and feature alignment. The approach includes three main modules： first， a feature representation module extracts more detailed features from text reports and chest images ； second， Knowledge-enhanced visual features incorporate prior medical knowledge， guiding the learning of visual features； and third， global-local feature alignment promotes semantic alignment between images， reports， and disease labels， thereby improving the accuracy and completeness of the generated reports.</p><p>Method First， chest X-ray images and text reports are input， and an image and text feature representation module is constructed， which includes visual and textual encoders to extract global and local features from the images and text， respectively. Then， a chest prior knowledge graph is introduced， enabling knowledge-enhanced visual feature learning through the encoding of pathological image knowledge， resulting in enhanced visual features after fusion. Finally， cross-attention is defined to align the global-local features of the image and text with the visual-disease labels across modalities， and multi-head attention is used in the encoder-decoder to generate accurate chest X-ray image reports.</p><p>Results The effectiveness of the proposed method was validated through comparative experiments on two challenging datasets， IU X-Ray and MIMIC-CXR. The results show that in the IU X-Ray dataset， the BLEU-1， 3， and 4 scores reached 0.505， 0.235， and 0.178， respectively， representing an improvement over most existing methods for the same task. This indicates that the proposed model offers certain advantages in text fluency and better focus on disease regions， generating correct label information， significantly improving various report metrics. The CIDEr and ROUGE-L scores are also comparable to those of other methods， demonstrating that knowledge enhancement and feature alignment positively impact the quality of the generated reports. On the MIMIC-CXR dataset， the BLEU-2 and 3 metrics increased by 0.4% and 1.2%， respectively， proving the robustness of the model， which maintains its advantages even when faced with more complex and diverse data. The CE metrics in the MIMIC-CXR dataset also showed improvement， with the precision metric reaching 0.428， recall reaching 0.343， and F1 reaching 0.360， indicating the model’s effectiveness in generating complete and consistent reports. Qualitative experiments show that the proposed method generates medical reports largely consistent with reference reports. Additionally， for abnormal images， the model accurately locates the abnormal regions， with the generated reports including not only descriptions of the disease conditions but also describe the lesions. The vocabulary used is professional and closely aligns with the ground truth reports， demonstrating that the knowledge enhancement contributes to the model's ability to generate professional and accurate text. Ablation experiments confirm that the inclusion of image and text feature representation， knowledge enhancement， and feature alignment modules improves report generation compared to the basic model. This suggests that incorporating external knowledge to the medical report generation model and fully utilizing the visual features of similar medical images enables the model learn sufficient visual features of medical images during the encoding stage， guiding the model to generate more complete and accurate medical reports during the decoding stage.</p><p>Conclusion The chest X-ray image report generation method proposed in this paper captures detailed features of both images and text， focus on the relationships between global-local features and disease categories， and enhance the alignment between images and text， enabling the generation of complete and accurate medical reports.</p>