arXiv:2604.08456cs.CVcs.CL2026-04被引 2

不训练模型,用不确定性引导视觉细节定位。

Entropy-Gradient Grounding: Training-Free Evidence Retrieval in Vision-Language Models

论文配图:Entropy-Gradient Grounding: Training-Free Evidence Retrieval in Vision-Language Models
图 1 · 摘自论文原文
  • 用视觉标记的熵梯度生成注意力图,无需额外检测器。
  • 在7个基准上提升准确率,高分辨率下效果更明显。
  • 适合需要解释性定位的细粒度视觉问答任务。

尽管预训练视觉语言模型进展迅速,但在依赖细微视觉差异或跨区域线索组合的任务中仍表现不佳,如文档理解与复合查询。本文提出一种无需训练的模型内接地方法,将定位问题转化为测试时证据检索:给定查询,模型应主动识别下一步关注位置以消除歧义。核心思想是利用模型输出的下一个词分布熵作为监督信号,反向传播至视觉标记嵌入,生成熵梯度相关性图,无需辅助检测器或注意力启发式规则。随后提取并排序多个连贯区域以支持多证据查询,并引入基于空间熵的迭代缩放-重定位机制,防止过度细化。在四个VLM架构的七个基准上实验表明,该方法显著优于现有方法,尤其在注重细节和高分辨率场景中提升最大,同时产生更具可解释性的证据定位结果。

原文摘要 · Abstract (English)

Despite rapid progress, pretrained vision-language models still struggle when answers depend on tiny visual details or on combining clues spread across multiple regions, as in documents and compositional queries. We address this by framing grounding as test-time evidence retrieval: given a query, the model should actively identify where to look next to resolve ambiguity. To this end, we propose a training-free, model-intrinsic grounding method that uses uncertainty as supervision. Specifically, we compute the entropy of the model's next-token distribution and backpropagate it to the visual token embeddings to obtain an entropy-gradient relevance map, without auxiliary detectors or attention-map heuristics. We then extract and rank multiple coherent regions to support multi-evidence queries, and introduce an iterative zoom-and-reground procedure with a spatial-entropy stopping rule to avoid over-refinement. Experiments on seven benchmarks across four VLM architectures demonstrate consistent improvements over existing methods, with the largest gains on detail-critical and high-resolution settings, while also producing more interpretable evidence localizations.

视觉语言模型零样本推理可解释性细粒度定位

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。