通过循环跨层推理减少视觉语言模型幻觉
HalluRNN: Mitigating Hallucinations via Recurrent Cross-Layer Reasoning in Large Vision-Language Models
- 设计共享的递归模块，持续优化各层隐藏状态
- 仅微调模块即在多基准上显著降低幻觉率
- 适合需要高可信输出的视觉问答与图像描述任务
尽管大型视觉语言模型在多种任务中表现卓越，但仍易产生文本合理但视觉无依据的幻觉。现有方法多依赖数据微调或特殊解码策略，成本高且需任务定制。本文提出架构级解决方案HalluRNN，通过递归跨层推理增强模型稳定性。核心是共享的双门控深度传播单元（DG-DPU），在各层间循环精炼隐藏状态，实现信息自适应传播，强化层间一致性，缓解表征漂移导致的幻觉。仅微调DG-DPU模块，即可在多个基准上实现强而稳健的表现。
原文摘要 · Abstract (English)
Though Large Vision-Language Models (LVLMs) have achieved remarkable performance across various tasks, they are still prone to hallucinations-generating outputs that are textually plausible but visually ungrounded. While prior approaches generally address this issue through data-centric fine-tuning or innovative decoding strategies, these methods often require substantial resources or task-specific configurations. In this work, we introduce an architecture-level solution, HalluRNN, which enhances model stability through recurrent cross-layer reasoning. Specifically, we propose a novel Dual-Gated Depth Propagation Unit (DG-DPU) module, which is shared across layers and recurrently refines hidden states. This allows for the adaptive propagation of information throughout the model, enforces consistency across layers, and mitigates hallucinations caused by representational drift. By fine-tuning only the DG-DPU module, HalluRNN achieves strong and robust performance across multiple benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。