Dual-modality visual feature flow for medical report generation

Abstract

• We propose a dual-modality visual feature flow for medical report generation, which extracts and aligns multi- angle medical image visual features with report text embeddings to enhance the model's inference capability. • We design a region-level feature extraction and enhancement module on top of grid-level features to enhance the visual encoding from both global and local perspectives, which is first attempt in deep medical report generation. • We align different visual features with medical report text embeddings to enhance the text inference capabilities of the model, considering different types of features. • Extensive comparison and ablation experiments demonstrate that DMVF outperforms state-of-the-art (SOTA) methods in terms of both quantitative and qualitative results. Medical report generation, a cross-modal task of generating medical text information, aiming to provide professional descriptions of medical images in clinical language. Despite some methods have made progress, there are still some limitations, including insufficient focus on lesion areas, omission of internal edge features, and difficulty in aligning cross-modal data. To address these issues, we propose Dual-Modality Visual Feature Flow (DMVF) for medical report generation. Firstly, we introduce region-level features based on grid-level features to enhance the method's ability to identify lesions and key areas. Then, we enhance two types of feature flows based on their attributes to prevent the loss of key information, respectively. Finally, we align visual mappings from different visual feature with report textual embeddings through a feature fusion module to perform cross-modal learning. Extensive experiments conducted on three benchmark datasets demonstrate that our approach outperforms the state-of-the-art methods in both natural language generation and clinical efficacy metrics.

Topics & Concepts