arXiv:2607.16841cs.CVcs.MM2026-07中稿 · ACM Multimedia 202…

通过视觉注意力重分配,减少大模型幻觉生成。

Look Clearly Before Answering: Mitigating Hallucinations in LVLMs via Saliency-Driven Perceptual Realignment

论文配图:Look Clearly Before Answering: Mitigating Hallucinations in LVLMs via Saliency-Driven Perceptual Realignment
图 1 · 摘自论文原文
  • 用显著性驱动注意力重分配,恢复被干扰的视觉信息。
  • 在生成过程中保持关键视觉特征,减少记忆偏差。
  • 无需训练,适合提升各类视觉语言模型可靠性。

大型视觉语言模型(LVLMs)在多模态理解方面表现出色,但容易产生与视觉证据不符的幻觉。现有方法多关注语言先验偏差或跨模态失衡,而对感知与记忆中视觉信息逐步退化的现象研究不足。本文提出无需训练的Saliency-Driven Perceptual Realignment(SDPR)框架,从生成全过程提升视觉意识一致性。首先,通过显著性驱动的注意力重分配,释放被非语义占位符劫持的注意力,恢复关键视觉证据;其次,识别键值缓存中的空间失真,提出显著性驱动缓存对齐,保留生成时查询相关的视觉特征;最后,引入先验约束对比解码,惩罚由主导语言先验引发的不忠实预测。大量实验表明,SDPR在多种LVLM架构上均优于现有方法,在幻觉检测与通用任务上表现更优,且无额外训练开销,运行延迟极低。代码已公开。

原文摘要 · Abstract (English)

Large vision-language models (LVLMs) have demonstrated remarkable capabilities in multimodal understanding. However, they remain prone to hallucinations, generating responses that are inconsistent with the visual evidence. Existing mitigation methods largely address language-prior bias or cross-modal imbalance, while progressive visual degradation across perception and memory remains underexplored. In this work, we propose Saliency-Driven Perceptual Realignment (SDPR), a training-free framework that mitigates the degradation of visual awareness throughout inference. Specifically, we first introduce saliency-driven attention redistribution to release attention hijacked by non-semantic sink tokens, thereby recovering critical visual evidence. Second, we identify spatial distortion in the KV cache and propose saliency-driven cache alignment to preserve query-relevant visual features during generation. Finally, we introduce prior-constrained contrastive decoding to penalize unfaithful predictions induced by dominant language priors. Our proposed SDPR is robust against hallucinations due to its holistic alignment of visual awareness across the entire generative trajectory. Extensive experiments across diverse LVLM architectures show that SDPR outperforms state-of-the-art methods on both hallucination and general-purpose benchmarks, requiring no additional training and incurring minimal runtime overhead. The code is available \href{https://github.com/PengSyuChen/SDPR}{\color{blue}{here}}.

视觉语言模型幻觉抑制注意力机制生成优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。