揭示视觉语言模型幻觉成因，通过修正错误生成提升可靠性
Revealing Perception and Generation Dynamics in LVLMs: Mitigating Hallucinations via Validated Dominance Correction
- 发现感知分三阶段：全局扫描→聚焦核心→探索补充区域
- 生成存在子主导累积现象，导致无依据的幻觉文本出现
- 提出验证主导性修正策略，显著减少幻觉且适配多模型
大型视觉语言模型（LVLMs）展现出强大能力，但幻觉问题仍持续存在。本文系统分析了LVLM内部视觉感知与标记生成的演变过程，揭示两个关键规律：其一，感知遵循三阶段GATE流程——早期层执行全局扫描，中间层聚焦并收紧核心内容，后期层探索补充区域；其二，生成呈现SAD（子主导累积至主导）模式，即幻觉标记源于缺乏注意力支持（视觉感知）或前馈网络支撑（内部知识）的子主导标记反复积累。基于此，提出VDC（验证主导性修正）策略，检测无支持标记并替换为经验证的主导标记，以提升输出可靠性。跨多个模型与基准的大量实验表明，VDC能显著缓解幻觉。
原文摘要 · Abstract (English)
Large Vision-Language Models (LVLMs) have shown remarkable capabilities, yet hallucinations remain a persistent challenge. This work presents a systematic analysis of the internal evolution of visual perception and token generation in LVLMs, revealing two key patterns. First, perception follows a three-stage GATE process: early layers perform a Global scan, intermediate layers Approach and Tighten on core content, and later layers Explore supplementary regions. Second, generation exhibits an SAD (Subdominant Accumulation to Dominant) pattern, where hallucinated tokens arise from the repeated accumulation of subdominant tokens lacking support from attention (visual perception) or feed-forward network (internal knowledge). Guided by these findings, we devise the VDC (Validated Dominance Correction) strategy, which detects unsupported tokens and replaces them with validated dominant ones to improve output reliability. Extensive experiments across multiple models and benchmarks confirm that VDC substantially mitigates hallucinations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。