用动态注意力累积抑制视觉大模型幻觉，无需重训且开销极小。
TARAC: Mitigating Hallucination in LVLMs via Temporal Attention Real-time Accumulative Connection
- 通过实时累积历史注意力，增强生成过程中的视觉连贯性。
- 在CHAIR上减少25.2%幻觉句，在MME上感知得分提升10.65点。
- 轻量级插件式设计，推理耗时仅增加约4%，适合部署优化。
大型视觉语言模型虽表现优异，但幻觉问题制约其实际应用。现有缓解方法常伴随高计算开销或需大量重训练。本文针对生成过程中视觉注意力衰减这一关键诱因，提出无需训练的TARAC框架——时间注意力实时累积连接。该方法受认知强化机制启发，动态累积并重新注入历史注意力，以维持视觉锚定。作为轻量级即插即用模块，TARAC在多个模型（如LLaVA、Qwen2-VL）和基准测试中显著优于现有方法。实验表明，其推理开销仅增加约4% TPOT，却在CHAIR上降低25.2%幻觉句子，在MME上提升感知得分10.65点，验证了其高效性与有效性。
原文摘要 · Abstract (English)
Large Vision-Language Models have demonstrated remarkable capabilities, yet they suffer from hallucinations that limit practical deployment. While various mitigation strategies exist, they often incur high computational overhead or require extensive retraining. In this paper, we address the issue of visual attention decay during generation, a key factor contributing to hallucinations. We propose Temporal Attention Real-time Accumulative Connection (TARAC), a novel training-free framework that dynamically accumulates and re-injects historical attention to sustain visual grounding. Inspired by cognitive reinforcement mechanisms, TARAC operates as a lightweight, plug-and-play module. Extensive experiments across diverse models (e.g., LLaVA, Qwen2-VL) and benchmarks demonstrate that TARAC significantly outperforms state-of-the-art methods. Remarkably, it achieves these gains with negligible inference overhead ($\sim$4\% TPOT increase), compared to the substantial costs of existing training-free baselines. Specifically, TARAC reduces hallucinated sentences by 25.2\% on CHAIR and improves Perception score by +10.65 on MME, validating its effectiveness and efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。