通过视觉聚光灯提升冻结多模态模型的证据感知能力
SPOT-E: Test-Time Entropy Shaping with Visual Spotlights for Frozen VLMs
- 用答案跨度熵作为反馈信号，动态生成问题相关的视觉聚焦区域
- 在多个基准上提升准确率，且在视觉干扰下保持更强鲁棒性
- 无需微调，可即插即用，适合希望增强现有模型推理能力的研究者
视觉-语言模型在依赖证据的任务中表现不佳，因其难以捕捉细小、局部的关键视觉证据，导致即使具备高层次推理能力也失效。现有推理时视觉干预方法大多为开环操作，缺乏验证所强调证据是否被实际使用的能力。本文将答案跨度预测熵作为模型内部反馈信号，发现简单的熵最小化存在歧义：低熵可能源于对证据的自信，也可能来自捷径坍缩。为此，提出低熵锚点与熵塑造目标，在降低答案不确定性的同时保留原有高置信度标记。基于此构建SPOT-E，一种无需训练的测试时方法，通过轻量级微调（基于组相对策略优化）生成问题相关的视觉聚光灯。在多个基准和不同VLM家族上均实现稳定提升，并在视觉退化条件下表现更稳健。代码已公开。
原文摘要 · Abstract (English)
Vision-language models (VLMs) often underperform on evidence intensive tasks because decisive visual evidence are small, localized, and easy to overlook, leading to failures in evidence readout even when high-level reasoning is intact. Prior inference-time visual interventions can improve grounding without retraining, but they are largely open-loop and lack a mechanism to verify whether highlighted evidence is actually used. We study answer-span prediction entropy as a model-internal feedback signal and show that naive entropy minimization is ambiguous, since low entropy may arise from evidence-grounded confidence or shortcut collapse. To resolve this ambiguity, we introduce low-entropy anchors and an entropy-shaping objective that reduces answer uncertainty while preserving baseline high-confidence tokens. We instantiate this principle in SPOT-E, a plug-and-play test-time method that produces question-conditioned spotlights, optimized per instance via light-weight tuning based on Group Relative Policy Optimization (GRPO). Across all benchmarks and different VLM families, SPOT-E yields consistent gains and improved robustness under visual corruptions. Code is publicly available at: https://github.com/YinBo0927/SPOT-E
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。