让AI看病时更专注影像细节,避免靠猜答案
InViC: Intent-aware Visual Cues for Medical Visual Question Answering
- 用关键词提取图像关键线索,强制模型看图答题
- 在多个医学问答数据集上提升准确率,效果稳定
- 适合需要高可信度医疗AI的临床场景使用
医学视觉问答(Med-VQA)旨在基于医学影像回答临床相关问题。现有多模态大模型常依赖语言先验或数据偏差进行捷径推理,忽视视觉证据,影响临床可靠性,尤其当细微影像发现决定诊断时。本文提出轻量级插件框架InViC,通过提示感知视觉线索(CTE)模块,将密集视觉特征压缩为K个与问题相关的线索令牌,作为结构化视觉中间表示注入大模型解码器,引导模型关注与意图一致的视觉信息。为防止跳过视觉输入,设计两阶段微调策略:第一阶段使用注意力掩码阻断模型直接访问原始视觉特征,强制所有视觉信息经由线索路径;第二阶段恢复标准因果注意力,训练模型联合利用视觉与线索令牌。在VQA-RAD、SLAKE和ImageCLEF VQA-Med 2019三个公开数据集上评估,InViC在多个代表性多模态大模型上均优于零样本推理和标准LoRA微调,证明引入提示感知视觉线索并结合瓶颈训练是提升可信医学视觉问答的有效策略。
原文摘要 · Abstract (English)
Medical visual question answering (Med-VQA) aims to answer clinically relevant questions grounded in medical images. However, existing multimodal large language models (MLLMs) often exhibit shortcut answering, producing plausible responses by exploiting language priors or dataset biases while insufficiently attending to visual evidence. This behavior undermines clinical reliability, especially when subtle imaging findings are decisive. We propose a lightweight plug-in framework, termed Intent-aware Visual Cues (InViC), to explicitly enhance image-based answer generation in medical VQA. InViC introduces a Cue Tokens Extraction (CTE) module that distills dense visual tokens into a compact set of K question-conditioned cue tokens, which serve as structured visual intermediaries injected into the LLM decoder to promote intent-aligned visual evidence. To discourage bypassing of visual information, we further design a two-stage fine-tuning strategy with a cue-bottleneck attention mask. In Stage I, we employ an attention mask to block the LLM's direct view of raw visual features, thereby funneling all visual evidence through the cue pathway. In Stage II, standard causal attention is restored to train the LLM to jointly exploit the visual and cue tokens. We evaluate InViC on three public Med-VQA benchmarks (VQA-RAD, SLAKE, and ImageCLEF VQA-Med 2019) across multiple representative MLLMs. InViC consistently improves over zero-shot inference and standard LoRA fine-tuning, demonstrating that intent-aware visual cues with bottlenecked training is a practical and effective strategy for improving trustworthy Med-VQA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。