用图结构融合CT图像与问题特征,提升医学影像问答准确性
Computed Tomography Visual Question Answering with Cross-modal Feature Graphing
- 构建跨模态图,将CT切片和问题词作为节点连接
- 在图上用注意力机制动态融合特征,生成精准答案
- 适合需要高精度医学影像理解的临床辅助场景
医学影像视觉问答旨在通过自动解析复杂影像数据来支持临床诊断。现有方法通常使用独立的视觉和文本编码器分别提取医学图像和临床问题特征,再进行融合生成答案。在计算机断层扫描(CT)中,这类方法沿用传统医学图像分析模式,但忽视了体积数据中的空间连续性与切片间相关性,导致回答碎片化、不精确。本文提出一种基于大语言模型(LLM)的新型框架,引入显著特征的图表示。不同于传统多模态编码策略,该方法构建一个融合视觉与文本特征的跨模态图,将每个CT切片和问题词元视为图中的节点,并利用注意力图卷积网络动态融合信息。最终聚合的图特征作为软提示,引导大语言模型生成准确回答。在M3D-VQA基准上的大量实验表明,本方法在多个评估指标上持续优于基线,展现出更强的推理能力。
原文摘要 · Abstract (English)
Visual question answering (VQA) in medical imaging aims to support clinical diagnosis by automatically interpreting complex imaging data in response to natural language queries. Existing studies typically rely on distinct visual and textual encoders to independently extract features from medical images and clinical questions, which are subsequently combined to generate answers. Specifically, in computed tomography (CT), such approaches are similar to the conventional practices in medical image analysis. However, these approaches pay less attention to the spatial continuity and inter-slice correlations in the volumetric CT data, leading to fragmented and imprecise responses. In this paper, we propose a novel large language model (LLM)-based framework enhanced by a graph representation of salient features. Different from conventional multimodal encoding strategies, our approach constructs a cross-modal graph integrating both visual and textual features, treating individual CT slices and question tokens as nodes within the graph. We further leverage an attentive graph convolutional network to dynamically fuse information within this structure. The resulting aggregated graph features then serve as a soft prompt to guide a large language model in generating accurate answers. Extensive experiments on the M3D-VQA benchmark demonstrate that our approach consistently outperforms baselines across multiple evaluation metrics, offering more robust reasoning capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。