通过可控提问方式,测出视觉语言模型依赖文字先验的问题。
Do Vision-Language Models See or Guess? Measuring and Reducing Textual-Prior Reliance with a Phrasing-Controlled Benchmark

- 设计四类提问变体,控制文本变量,分离图像与文字影响。
- 所有模型在最难提问下性能下降,开源模型降幅最大,最低仅1%准确率。
- 引入无图像消融和人类重标注,验证模型真实依赖图像的能力。
视觉语言模型(VLMs)在实际应用中需基于图像内容作答,但常依赖文本先验——即问题表述与记忆的世界知识,而非图像本身,导致基准分数虚高且答案缺乏依据。现有评测通常每图仅配一个固定问题,难以分离该行为。为此,我们构建了一个包含540张图像的基准,覆盖六类推理任务,对每张图生成四个问题变体,使提问方式成为唯一控制变量。最困难的变体直接从图像描述生成,以最小化文本泄露。评估了十一个VLM,涵盖小规模开源模型到大型闭源系统:所有模型在最难变体上性能下降,开源模型降幅最大,无图像时其准确率仅1%~9%。进一步分析包括LLM评分难度、低文本相似度、人类重标注,均证实模型存在真实图像依赖性。使用与变体构建方式匹配的上下文示例可恢复最高准确率;对小型模型进行GRPO后训练,在所有变体上均获得稳定提升,并泛化至未见分布数据集。文本先验依赖可测量,且部分可通过训练缓解。
原文摘要 · Abstract (English)
Vision-language models (VLMs) are increasingly deployed where answers must follow from what is in the image, yet they often answer from textual priors, the question's phrasing together with memorized world knowledge, rather than from the image itself, which inflates benchmark scores and yields confident but ungrounded answers. Existing benchmarks rarely isolate this behavior, since each image is usually paired with a single fixed question. To measure the reliance, we build a 540-image benchmark across six reasoning categories and generate four question variants over the same images, so that phrasing rather than image content is the controlled variable. The hardest variant is written directly from the image to minimize text leakage. We benchmark eleven VLMs spanning small open-weight models to large closed-source systems: every model degrades on the hardest variant, and open models fall furthest. Our central diagnostic is a no-image ablation, which collapses the open-weight models to their text-only floor (1 to 9 percent). Three further analyses, LLM-rated difficulty, low base-to-final textual similarity, and human re-annotation, corroborate genuine image-dependence. In-context exemplars that match how a variant was built recover the most accuracy, and GRPO post-training of a small VLM yields consistent gains across all four variants that transfer to a held-out out-of-distribution set. Textual-prior reliance is measurable and partly trainable away.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。