让视觉模型不瞎编，通过保留并重用视觉证据提升回答准确性
EviAnchor: Mitigating Hallucinations in Large Vision-Language Models via Regional Visual Evidence Compensation
- 在生成过程全程保留视觉信息，通过区域锚点聚合视觉特征
- 实验显示在POPE、CHAIR等数据集上视觉定位准确率显著提升
- 无需训练、仅改推理流程，适合快速部署到现有大模型
大型视觉语言模型常生成与图像无关的内容。初步实验表明，视觉证据主要在解码器的早期至中期层被纳入答案表示，但其直接影响随层数加深而减弱。这说明早期获取的视觉信息在后续生成中未被充分使用。针对此问题，我们提出EviAnchor——一种无需训练、单分支推理的框架，可在生成全过程保持并重激活视觉证据。EviAnchor引入区域证据锚点（REA）槽，将密集视觉标记逐步聚合为结构化空间表示，并通过决策条件化的证据路由机制增强当前决策状态对这些视觉锚点的访问能力，缓解对文本上下文的过度依赖。最终，模型恢复原有Transformer计算，将检索到的视觉证据与问题语义及生成历史融合。在POPE、CHAIR和MMHal-Bench上的实验表明，该方法在视觉定位任务上持续提升性能。
原文摘要 · Abstract (English)
Large vision-language models (LVLMs) frequently generate content unsupported by visual inputs. Preliminary experiments show that visual evidence is primarily incorporated into answer-side representations in early-to-middle decoder layers, while its direct influence progressively weakens in later layers. This attenuation suggests that visual evidence acquired earlier may be insufficiently utilized during subsequent generation. Based on this observation, we propose EviAnchor, a training-free and single-branch inference framework that preserves and reactivates visual evidence throughout generation. EviAnchor introduces Regional Evidence Anchor (REA) slots to progressively aggregate dense visual tokens into spatially structured representations. It then strengthens the current decision state's access to these visual anchors through decision-conditioned evidence routing, mitigating excessive dependence on textual context. Finally, the model resumes its native Transformer computation to integrate the retrieved visual evidence with question semantics and generation history. Experiments across POPE, CHAIR, and MMHal-Bench demonstrate consistent improvements in visual grounding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。