测试视觉语言模型对图像可见性的判断能力，区分可知与不可知情况。
VB: Visibility Benchmark for Visibility and Perspective Reasoning in Images
- 设计100组图像+文本最小编辑对，构建可验证的可见性判断任务
- 模型需判断可见性并合理弃权，正确率最高达0.728（GPT-4o/Gemini）
- 强调判断依据和信心校准，适合评估模型推理可靠性
我们提出VB基准，测试视觉语言模型能否判断照片中某物是否可见，并在人类也无法确定时选择弃权。每项包含一张图片和一个简单的“是/否”可见性陈述，模型需输出VISIBLY_TRUE、VISIBLY_FALSE或ABSTAIN，并附上置信度分数。项目按2×2设计分为100个类别，通过最小图像编辑与最小文本编辑组合生成300个评测单元。与以往不可回答VQA基准不同，VB不仅判断问题是否不可答，还通过关联具体可见性因素的归因代码解释原因，并利用受控最小编辑验证模型判断仅在证据变化时才改变。我们采用置信度感知准确率（CAA）、最小编辑翻转率（MEFR）、置信度排序选择性预测（SelRank）及二阶视角推理（ToMAcc）四项指标评估模型，所有核心数据均基于严格XOR子集（每类3个单元，每模型300项）。评估涵盖9个模型，包括旗舰级闭源系统与8B至12B参数的开源模型。GPT-4o与Gemini 3.1 Pro以0.728和0.727并列最佳，次为Gemini 2.5 Pro（0.678）。最佳开源模型Gemma 3 12B得分为0.505，超过某些旧版闭源系统。六种模型中，文本翻转鲁棒性优于图像翻转，且置信度校准差异显著：GPT-4o与Gemini 2.5 Pro准确率相近，但在选择性预测质量上差距明显。
原文摘要 · Abstract (English)
We present VB, a benchmark that tests whether vision-language models can determine what is and is not visible in a photograph, and abstain when a human viewer cannot reliably answer. Each item pairs a single photo with a short yes/no visibility claim; the model must output VISIBLY_TRUE, VISIBLY_FALSE, or ABSTAIN, together with a confidence score. Items are organized into 100 families using a 2x2 design that crosses a minimal image edit with a minimal text edit, yielding 300 headline evaluation cells. Unlike prior unanswerable-VQA benchmarks, VB tests not only whether a question is unanswerable but why (via reason codes tied to specific visibility factors), and uses controlled minimal edits to verify that model judgments change when and only when the underlying evidence changes. We score models on confidence-aware accuracy with abstention (CAA), minimal-edit flip rate (MEFR), confidence-ranked selective prediction (SelRank), and second-order perspective reasoning (ToMAcc); all headline numbers are computed on the strict XOR subset (three cells per family, 300 scored items per model). We evaluate nine models spanning flagship and prior-generation closed-source systems, and open-source models from 8B to 12B parameters. GPT-4o and Gemini 3.1 Pro effectively tie for the best composite score (0.728 and 0.727), followed by Gemini 2.5 Pro (0.678). The best open-source model, Gemma 3 12B (0.505), surpasses one prior-generation closed-source system. Text-flip robustness exceeds image-flip robustness for six of nine models, and confidence calibration varies substantially: GPT-4o and Gemini 2.5 Pro achieve similar accuracy yet differ sharply in selective prediction quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。