通过减少无关图像标记，提升视觉语言模型的生成可靠性。
MINT: Mitigating Hallucinations in Large Vision-Language Models via Token Reduction
- 动态屏蔽无关图像标记，增强模型局部感知能力。
- 在减少图像标记的同时，使模型多关注5%的关键视觉点。
- 无需训练，显著降低幻觉率，提升生成准确性。
幻觉是大型视觉语言模型（LVLM）在高可靠性领域应用中的长期难题。现有方法多依赖数据标注或训练策略改进，却忽视了大语言模型自身的问题。本文深入研究了LVLM解码过程中的注意力机制，发现其层级架构中存在普遍的注意力冗余，表现为深层过度处理图像及大量非必要图像标记。基于此，提出MINT——一种无需训练的解码策略，通过动态屏蔽无关图像标记并采用对比解码，引导模型聚焦关键视觉区域。实验表明，该方法在多个公开基准上使由分心感知导致的幻觉减少4%，且在减少图像标记的情况下，仍能提升5%的视觉感知精度。
原文摘要 · Abstract (English)
Hallucination has been a long-standing and inevitable problem that hinders the application of Large Vision-Language Models (LVLMs) in domains that require high reliability. Various methods focus on improvement depending on data annotations or training strategies, yet place less emphasis on LLM's inherent problems. To fill this gap, we delve into the attention mechanism of the decoding process in the LVLM. Intriguingly, our investigation uncovers the prevalent attention redundancy within the hierarchical architecture of the LVLM, manifesting as overextended image processing in deep layers and an overabundance of non-essential image tokens. Stemming from the observation, we thus propose MINT, a novel training-free decoding strategy, MItigating hallucinations via tokeN reducTion. Specifically, we dynamically intensify the LVLM's local perception capability by masking its attention to irrelevant image tokens. In addition, we use contrastive decoding that pushes the model to focus more on those key image regions. Our full method aims to guide the model in concentrating more on key visual elements during generation. Extensive experimental results on several popular public benchmarks show that our approach achieves a 4% improvement in mitigating hallucinations caused by distracted perception compared to original models. Meanwhile, our approach is demonstrated to make the model perceive 5% more visual points even though we reduce a suite of image tokens.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。