arXiv:2604.25855cs.CVcs.AI2026-04

通过视觉证据评分实现选择性预测,提升模型在复杂场景下的可靠性和覆盖范围。

SIEVES: Selective Prediction Generalizes through Visual Evidence Scoring

论文配图:SIEVES: Selective Prediction Generalizes through Visual Evidence Scoring
图 1 · 摘自论文原文
  • 基于视觉定位质量评分设计选择性预测器,无需模型内部参数。
  • 在多个外部数据集上覆盖率最高提升三倍,且误差率可控。
  • 适用于闭源模型,可跨模型迁移,适合实际部署场景。

多模态大语言模型在视觉语言任务中表现持续提升。尽管传统视觉问答(VQA)基准接近饱和,真实世界中的可靠部署仍需满足低错误容忍度的分布外(OOD)场景要求。选择性预测旨在提高覆盖率(即系统能回答的输入比例),同时控制在用户定义的风险水平内。现有方法依赖模型内部信号(如logits、隐藏表示)估算置信度,但无法用于前沿闭源模型。为此,我们提出SIEVES:要求推理模型在作答时提供局部化视觉证据,并设计一个仅使用输入输出的显式证据质量评估器。实验表明,SIEVES在多个挑战性OOD基准(V* Bench、HR-Bench-8k、MME-RealWorld-Lite、VizWiz、AdVQA)上,覆盖率最高提升三倍,优于非基于定位的基线。此外,该方法可无权重访问地适配闭源模型(如o3、Gemini-3-Pro),带来超越准确率提升的覆盖率增益。SIEVES在所有测试的基准和模型(Pixel-Reasoner、o3、Gemini-3-Pro)间均表现泛化,无需特定训练或微调。代码已公开于https://github.com/hector-gr/SIEVES。

原文摘要 · Abstract (English)

Multimodal large language models (MLLMs) achieve ever-stronger performance on visual-language tasks. Even as traditional visual question answering (VQA) benchmarks approach saturation, reliable deployment requires satisfying low error tolerances in real-world, out-of-distribution (OOD) scenarios. Precisely, selective prediction aims to improve coverage, i.e. the share of inputs the system answers, while adhering to a user-defined risk level. This is typically achieved by assigning a confidence score to each answer and abstaining on those that fall below a certain threshold. Existing selective prediction methods estimate implicit confidence scores, relying on model internal signals like logits or hidden representations, which are not available for frontier closed-sourced models. To enable reliable generalization in VQA, we require reasoner models to produce localized visual evidence while answering, and design a selector that explicitly learns to estimate the quality of the localization provided by the reasoner using only model inputs and outputs. We show that SIEVES (Selective Prediction through Visual Evidence Scoring) improves coverage by up to three times on challenging OOD benchmarks (V* Bench, HR-Bench-8k, MME-RealWorld-Lite, VizWiz, and AdVQA), compared to non-grounding baselines. Beyond better generalization to OOD tasks, the design of the SIEVES selector enables transfer to proprietary reasoners without access to their weights or logits, such as o3 and Gemini-3-Pro, providing coverage boosts beyond those attributable to accuracy alone. We highlight that SIEVES generalizes across all tested OOD benchmarks and reasoner models (Pixel-Reasoner, o3, and Gemini-3-Pro), without benchmark- or reasoner-specific training or adaptation. Code is publicly available at https://github.com/hector-gr/SIEVES .

视觉问答选择性预测闭源模型泛化能力

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。