提出新评估框架与训练方法，让医疗视觉语言模型真正依赖图像做判断。
Do Medical Vision Language Models Actually See? A Counterfactual Grounding Framework and Hard-Negative Contrastive Training for Visually-Reliant Medical VLMs
- 用可控替代图分离视觉与文本影响，量化模型对图像的依赖程度。
- 新模型CORAL在4个医学数据集上准确率提升6.7个百分点，幻觉率降低8.0个百分点。
- 适合关注医疗AI可解释性与视觉推理能力的研究者使用。
大型视觉语言模型（VLMs）在医学问答中表现优异，但其是否真正基于视觉证据推理仍不明。本文提出反事实评估框架，通过替换输入图像为空白、像素打乱、缺失图像及CLIP检索的难负样本，分离视觉与文本贡献，并定义视觉依赖度评分（VRS）和视觉幻觉率（VHR）。进一步提出CORAL（对比检索增强学习），一个基于Qwen2.5-VL-7B的70亿参数LoRA微调模型，采用对比接地目标（CGO），惩罚在难负样本替换下答案不变的情况。在四个闭合形式医学VQA基准（PathVQA、PMC-VQA、SLAKE、VQA-RAD；n=400）上的配对控制评估显示，CORAL相比基线模型宏准确率提升6.7个百分点（P(Delta>0)=0.988），VHR降低8.0个百分点（P<0.001）；两种MedVLThinker RL变体均未显著提升。跨领域诊断表明，图像替换在医学任务中损失≤6.5个百分点，而在通用任务中损失达48–61个百分点，凸显了当前接地差距。作者公开框架、代码与模型权重，支持可复现的医疗VLM接地审计，并坦诚讨论评估局限，包括训练/评测集重叠及次级指标功效不足。
原文摘要 · Abstract (English)
Large vision language models (VLMs) report strong accuracy on medical question-answering, yet it remains unclear whether they reason from visual evidence or exploit textual shortcuts. We introduce a counterfactual evaluation framework that decouples visual and textual contributions by substituting input images with controlled surrogates blank, pixel-shuffled, image-absent, and CLIP-retrieved hard negatives and derive a suite of grounding metrics including the Visual Reliance Score (VRS) and Visual Hallucination Rate (VHR). We further introduce CORAL (COntrastive Retrieval-Augmented Learning), a 7B-parameter LoRA fine-tune of Qwen2.5-VL-7B trained with a Contrastive Grounding Objective (CGO) that penalises answer invariance under hard-negative image swaps. On a paired controlled evaluation across four closed-form medical VQA benchmarks (PathVQA, PMC-VQA, SLAKE, VQA-RAD; n=400 total), CORAL improves macro accuracy by +6.7 pp (P(Delta>0)=0.988) and reduces VHR by 8.0 pp (P<0.001) over the matched Qwen2.5-VL-7B base; neither MedVLThinker RL variant achieves a significant gain on either metric. Cross-domain diagnostics further reveal that image substitution costs only <=6.5 pp on medical benchmarks versus 48-61 pp on general-domain tasks, situating the grounding gap that CGO targets. We discuss evaluation limitations openly including train/eval benchmark overlap and underpowered secondary metrics and release our framework, training code, and model weights to support reproducible grounding audits of medical VLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。