arXiv:2605.20158cs.CVcs.AI2026-05

提出MedFocus方法,让医学视觉语言模型的推理更可信。

Rethinking Visual Attribution for Chest X-ray Reasoning in Large Vision Language Models

论文配图:Rethinking Visual Attribution for Chest X-ray Reasoning in Large Vision Language Models
图 1 · 摘自论文原文
  • 用反事实编辑构建因果评估框架,验证视觉归因是否真实。
  • 11种归因方法在6个模型上表现差,多数无法找到真正证据。
  • 新方法通过解剖概念定位关键区域,适合临床可解释性研究。

大型视觉语言模型(LVLMs)在医疗应用中前景广阔,但其回答难以基于真实视觉证据,影响临床信任。现有视觉归因方法虽广泛使用,却缺乏对模型内部推理依据的验证,因缺少真实推理标注。本文针对胸片(CXR)推理问题,构建因果评估框架,仅保留专家标注区域经反事实编辑证实为模型预测因果因素的样本。基于该框架,在11种归因方法、6个开源LVLMs及两种输出模式(直接回答与分步推理)下测试发现,多数归因方法未能识别模型实际依赖的视觉证据。为此,提出MedFocus——一种基于概念的归因方法,利用非平衡最优传输定位临床相关解剖区域,并通过定向干预测量其对输出的因果影响。该方法生成空间、概念层级和词元级归因,显著优于现有方法,推动医学LVLM可解释性进步。数据与代码已公开于https://github.com/gzxiong/medfocus/。

原文摘要 · Abstract (English)

Large Vision Language Models (LVLMs) show promise in medical applications, but their inability to faithfully ground responses in visual evidence raises serious concerns about clinical trustworthiness. While visual attribution methods are widely used to explain LVLM predictions, whether these explanations actually reflect the visual evidence underlying the model's decision is largely unverified, since ground-truth annotations for internal model reasoning are typically unavailable. We address this question for chest X-ray (CXR) reasoning by developing a causal evaluation framework that retains only CXR-VQA samples for which the expert-annotated region is verified, via counterfactual editing, to be causally responsible for the model's prediction. Using this framework across 11 attribution methods, six open-source LVLMs, and two output modes (direct answer and step-by-step reasoning), we find that existing attribution methods often fail to identify the evidence used by LVLMs. To address this failure, we propose MedFocus, a concept-based attribution method that localizes clinically meaningful anatomical regions via unbalanced optimal transport and measures their causal effect on model outputs through targeted interventions. MedFocus produces spatial, concept-level, and token-level attributions and substantially outperforms prior methods, taking a step toward more trustworthy attribution for medical LVLMs. Our data and code are available at https://github.com/gzxiong/medfocus/.

医学图像归因分析视觉语言模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。