医学视觉问答中，定位证据反而降低准确率，因裁剪会丢失全局信息。
Why Does Grounding Hurt Medical VQA? Benchmarking, Diagnosis, and Fine-Tuning of Vision-Language Models
- 用边界框裁剪图像会破坏模型依赖的全局上下文，导致答案错误。
- 所有模型在医学目标定位上的平均交并比仅0.05–0.24，远低于实用标准。
- 加入少量定位监督可恢复定位能力，同时保持高问答准确率，适合临床应用。
视觉语言模型（VLMs）在医学视觉问答（Med-VQA）中应用日益广泛，但其能否精确定位回答依据——这对临床可审计性至关重要——尚未得到充分研究。我们在VQA-RAD和SLAKE数据集上评估了四种前沿VLM（GPT-5.1、GPT-5.5、Gemini-2.5-Pro、Gemini-3-Flash）、两个专用医疗VLM（Lingshu、MedGemma）以及一个开放词汇检测器（Grounding DINO）的VQA推理与视觉定位能力。结果显示：第一，无一模型能有效定位医学目标，所有系统在SLAKE定位测试集上的平均交并比（IoU）仅为0.05–0.24，仅略高于中心框基线（0.10），且[email protected]低于20%。第二，违背“先定位再回答”的直觉，即使使用完美真值边界框进行裁剪，所有模型的闭合式问答准确率仍下降0.9–18.0个百分点——说明问题不在于感知误差，而在于裁剪本身破坏了模型所依赖的全局上下文。第三，通过微调证明二者可共存：仅对答案进行监督时，Qwen-2.5-VL-7B完全丧失生成定位框的能力（0/418可解析框）；而加入少量定位监督后，定位性能提升至0.36 IoU，超过所有零样本模型，且问答准确率不受影响。
原文摘要 · Abstract (English)
Vision-language models (VLMs) are increasingly applied to medical visual question answering (Med-VQA), yet whether they can \emph{localize} the evidence behind their answers---a prerequisite for clinical auditability---is poorly characterized. We separately evaluate VQA reasoning and visual grounding for four recent frontier VLMs (GPT-5.1, GPT-5.5, Gemini-2.5-Pro, Gemini-3-Flash), two domain-specific medical VLMs (Lingshu, MedGemma), and a dedicated open-vocabulary detector (Grounding DINO) on VQA-RAD and SLAKE. Two findings challenge the intuition that ``add grounding to improve VQA.'' First, \textbf{no model localizes medical targets well}: every off-the-shelf system---frontier, medical-specialized, or dedicated detector---scores mean IoU 0.05--0.24 on our SLAKE grounding split, at or barely above a trivial center-box baseline (0.10), with [email protected] below 20\%. Second, and counter to the common ``localize-then-answer'' paradigm, \textbf{cropping to a bounding box degrades VQA even when the box is a perfect oracle}: on the matched subset where oracle ground-truth boxes are applied, GT-grounding \emph{lowers} closed-ended accuracy for every model (by 0.9--18.0 points versus using the full image)---consistent with the crop discarding global context the model relies on. Because the oracle box removes localization error by construction, the problem is not that perception is a recoverable bottleneck, but that grounding-by-cropping is itself the wrong interface. Finally, we show constructively that the two channels need not conflict: supervised fine-tuning of Qwen-2.5-VL-7B on answers \emph{alone} silently destroys box-evidence emission (0/418 parseable boxes), whereas mixing in a small amount of grounding supervision restores localization to 0.36 IoU---above every zero-shot model---while preserving answer accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。