arXiv:2602.13600cs.CV2026-02被引 2

通过动态调整注意力增强,有效减少大模型生成中的幻觉问题。

SAVAA: Mitigating Hallucinations in LVLMs via Step-wise Adaptive Visual Attention Amplification

  • 根据每步生成风险动态调整视觉注意力放大强度
  • 在多个主流模型上显著降低幻觉率,提升生成准确性
  • 适合需要高可信度图文生成的场景或研究者使用

近期无需训练的幻觉缓解方法通过单次前向传播中放大视觉标记注意力来抑制大视觉语言模型(LVLMs)的幻觉。我们称此范式为视觉注意力放大(VAA)。本文发现现有VAA方法因在生成各步使用固定放大因子而存在双重失效:某些步骤过弱,无法消除幻觉;某些步骤过强,反而引入新幻觉。为此,我们提出分步自适应视觉注意力放大(SAVAA),通过估计每步生成令牌的幻觉风险,动态调节下一生成步的注意力放大强度。具体地,引入轻量级幻觉风险评估器——视觉锚定熵(VGE),在预测熵基础上融合视觉锚定信息,对不确定、图像关联弱或两者兼具的标记赋予更高风险评分。基于VGE评估结果,SAVAA为下一流程校准VAA系数,高风险步骤强放大,低风险步骤弱放大。在LLaVA-NeXT-7B、Qwen3-VL-8B和InternVL3.5-8B等模型上,SAVAA在CHAIR、SHR和AMBER等生成幻觉基准测试中显著优于基线方法。代码已开源:https://github.com/JiachengZ01/SAVVA。

原文摘要 · Abstract (English)

A line of recent training-free methods for mitigating hallucinations in large vision-language models (LVLMs) operates by amplifying attention to visual tokens during autoregressive generation within a single forward pass. We refer to this paradigm as visual attention amplification (VAA). In this paper, we identify a dual failure pattern in existing VAA methods caused by their use of a fixed amplification factor across generation steps: it can be too weak at some steps, leaving hallucinations unresolved, while too strong at others, introducing new hallucinations. Motivated by this finding, we propose Step-wise Adaptive Visual Attention Amplification (SAVAA), a new VAA framework that estimates hallucination risk for each generated token and uses the estimated risk to adaptively amplify visual attention at the next generation step. Specifically, we introduce Visual Grounding Entropy (VGE), a lightweight hallucination-risk estimator that augments predictive entropy with visual grounding, assigning higher risk to tokens that are uncertain, weakly grounded in the image, or both. Guided by VGE, SAVAA uses the estimated risk to calibrate the VAA factor for the next generation step, applying stronger amplification to higher-risk steps and weaker amplification to lower-risk steps. Across LLaVA-NeXT-7B, Qwen3-VL-8B, and InternVL3.5-8B, SAVAA significantly outperforms baseline methods on generative hallucination benchmarks such as CHAIR, SHR and AMBER. Code is available at: https://github.com/JiachengZ01/SAVVA.

幻觉缓解视觉注意力生成质量大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。