用层间注意力实现模型内部自我纠错，减少视觉语言模型幻觉
Self-Correction Inside the Model: Leveraging Layer Attention to Mitigate Hallucinations in Large Vision Language Models
- 通过层间注意力机制让模型在生成时自我修正隐藏状态
- 仅增加0.2M参数，显著提升多个基准上的视觉一致性
- 适合追求高精度生成的先进视觉语言模型研究者使用
尽管大型视觉语言模型（LVLM）已取得显著进展，但生成内容与视觉输入不符的幻觉问题仍存在。随着模型能力增强，以往常见的语言偏差和过度思考现象变得不一致，导致原有缓解方法效果下降。本文提出一种内生自校正机制（ICLA），直接作用于生成过程中的隐藏状态。各层通过对角跨层注意力机制，从所有前序层中选择性检索信息，实现无需外部信号的自我精炼。在LLaVA1.5-7B和Qwen2.5-VL-7B上，分别仅引入0.2M和0.1M额外参数，该方法在多个幻觉评测基准上持续提升视觉接地性能，验证了其对更先进LVLM的有效性。
原文摘要 · Abstract (English)
Although Large Vision-Language Models (LVLMs) have made substantial progress, hallucination, where generated text is not grounded in the visual input, remains a challenge. As LVLMs become stronger, previously reported hallucination patterns, such as linguistic bias and overthinking phenomenon, become far less consistent, making the corresponding mitigation techniques substantially less effective. In this paper, we introduce an Internal self-Correction mechanism utilizing Layer Attention (ICLA) that operates directly on hidden states during generation. Each layer selectively retrieves information from all preceding layers through a diagonal cross-layer attention mechanism, enabling self-refinement without any external correction signals. With introducing and training only 0.2M and 0.1M additional parameters on LLaVA1.5-7B and Qwen2.5-VL-7B, \ours consistently improves visual grounding across multiple hallucination benchmarks, demonstrating its effectiveness for more advanced LVLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。