通过重分配视觉注意力提升模型真实性，有效减少幻觉。
Beyond Dominant Patches: Spatial Credit Redistribution For Grounded Vision-Language Models
- 提出空间信用重分配方法，不修改权重，在推理时恢复被压制的上下文。
- 在多个基准上降低幻觉率4.6-6.0个百分点，关键任务下降41-51%。
- 轻量级设计，每令牌增加延迟不足0.5毫秒，适合实际部署。
视觉语言模型常会生成输入图像中不存在的对象。我们识别出一个原因——空间信用坍缩：早期变换器层中，隐藏状态过度集中在少数视觉补丁上，抑制了周围上下文信息，导致对语言先验的依赖增强。在七种模型中，视觉注意力熵与幻觉率呈强负相关（r = -0.65, p < 0.001），表明空间信用多样性下降加剧幻觉。为此，我们提出空间信用重分配（SCR），一种无需训练的推理时方法。SCR采用轻量级两阶段过程：诊断阶段识别前K个高注意力源补丁及其邻域；重分配阶段将每个源补丁按1/λ（约0.91）缩放，并将其隐藏状态加权拷贝注入邻近补丁，恢复被压制的视觉上下文，且不修改模型权重。因诊断阶段仅需每图像执行一次并复用于输出序列，额外延迟可忽略（100词响应下每令牌<0.5毫秒）。我们在四个模型家族（Chameleon、LLaVA-1.5、Qwen-VL/Qwen2-VL、InternVL2）的七种配置上，于五个基准（POPE、CHAIR、MME、HallusionBench、AMBER）测试。SCR使POPE-Adversarial幻觉率降低4.6-6.0个百分点，CHAIR-s下降41-51%，同时保持生成质量（CIDEr下降≤0.8）。相比OPERA、VCD、OA-VCD、DoLa、VLI、SID、CRoPS等现有推理时方法，SCR在幻觉抑制、生成质量与延迟间取得更优平衡。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) often hallucinate objects that are not present in the input image. We identify a contributing cause of this behavior, which we term spatial credit collapse: in early transformer layers, hidden-state activation concentrates on a small number of visual patches, suppressing surrounding contextual evidence and increasing reliance on language priors. Across seven models we observe a strong correlation between visual attention entropy and hallucination rate (r = -0.65, p < 0.001), suggesting that reduced spatial credit diversity contributes to hallucination. To address this issue we propose Spatial Credit Redistribution (SCR), a training-free inference-time method. SCR uses a lightweight two-pass procedure. A diagnostic pass identifies the top-K high-attention source patches and their spatial neighbors. A redistribution pass then scales each source by 1/lambda (~0.91) and injects a (lambda - 1) weighted copy of its hidden state into neighboring patches, restoring suppressed visual context without modifying model weights. Because the diagnostic pass is performed once per image and reused across the output sequence, the added latency is negligible (<0.5 ms per token for 100-token responses). We evaluate SCR across seven model configurations from four VLM families (Chameleon, LLaVA-1.5, Qwen-VL/Qwen2-VL, and InternVL2) on five benchmarks: POPE, CHAIR, MME, HallusionBench, and AMBER. SCR reduces POPE-Adversarial hallucination by 4.6-6.0 percentage points and CHAIR-s by 41-51 percent while preserving caption quality (CIDEr drop <=0.8). Compared with prior inference-time methods including OPERA, VCD, OA-VCD, DoLa, VLI, SID, and CRoPS, SCR achieves a better trade-off between hallucination reduction, generation quality, and latency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。