通过动态补偿跨层与时间漂移，降低视觉语言模型幻觉率。
Dynamic Alignment Compensation for Hallucination Mitigation in Large Vision-Language Models
- 在推理时检测表示漂移，分层与序列级轻量补偿
- 九个基准测试中幻觉率显著下降，性能保持稳定
- 无需训练，适配多种主流视觉语言模型
大型视觉语言模型（LVLM）仍易产生与多模态输入无关或不一致的幻觉。现有缓解方法主要依赖外部监督、输出校准或注意力调节，未深入探索自回归生成过程中的内部表征动态。我们发现推理阶段存在一种失败模式：跨模态表征在解码器各层间退化，并随生成步骤发生漂移，导致标记预测不稳定，增加幻觉风险。为此提出训练无关的推理时方法——动态对齐补偿（DAC），可检测表征发散并选择性施加轻量残差补偿。DAC结合层间语义补偿以缓解层间退化，以及序列语义修正以约束时间漂移。在多个LVLM主干网络上，九个聚焦幻觉和通用的多模态基准测试均表明，DAC能持续降低幻觉率，同时保持优异整体性能。
原文摘要 · Abstract (English)
Large Vision-Language Models (LVLMs) remain prone to hallucinations, producing responses that are irrelevant or inconsistent with the multimodal input. Existing mitigation methods mainly rely on external supervision, output calibration, or attention regulation, leaving the internal representation dynamics of autoregressive generation underexplored. We identify an inference-time failure mode in which cross-modal representations degrade across decoder layers and drift across generation steps, destabilizing token prediction and increasing hallucination risk. We propose \emph{Dynamic Alignment Compensation} (DAC), a training-free inference-time method that detects representation divergence and selectively applies lightweight residual compensation. DAC combines Layer-wise Semantic Compensation to mitigate inter-layer degradation with Sequential Semantic Correction to constrain temporal drift. Experiments on nine hallucination-focused and general-purpose multimodal benchmarks across multiple LVLM backbones show that DAC consistently reduces hallucinations while maintaining strong overall performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。