用视觉注意力校准语言生成,减少幻觉错误。
Seeing to Ground: Visual Attention for Hallucination-Resilient MDLLMs
- 通过分析跨注意力分布熵值,识别并修正语言主导的错误优化
- 在MMM U-val和HallusionBench上分别提升8.59%和7.75%
- 无需训练即可部署,适合追求稳定多模态输出的场景
多模态扩散大语言模型(MDLLMs)通过并行掩码解码实现高并发生成,但架构仍易产生多模态幻觉。这一结构性缺陷源于算法漏洞:解码器仅依据文本似然排序候选词,未验证局部视觉支持。我们指出,这种纯语言排名导致目标不匹配,语言概率质量成为非预期任务的误标代理。因此,我们将幻觉重新解释为局部优化误差——解码器利用语言捷径最大化代理得分,牺牲视觉对齐。为此,我们提出VISAGE,一种无需训练的推理阶段目标校准框架。VISAGE通过量化跨注意力分布的空间熵来估计代理偏差,强制注意力头间空间一致性,惩罚均匀分布,重新排序词汇承诺以偏好视觉对齐结果。我们提供了分析性稳定性保证,证明在估计误差下目标损失保持有界。在多个幻觉敏感与通用基准上评估显示,该框架具备鲁棒性,在MMM U-val上相对提升8.59%,在HallusionBench上提升7.75%。
原文摘要 · Abstract (English)
Multimodal Diffusion Large Language Models (MDLLMs) achieve high-concurrency generation through parallel masked decoding, yet the architectures remain prone to multimodal hallucinations. This structural vulnerability stems from an algorithmic flaw: the decoder ranks candidate tokens based on textual likelihood without verifying localized visual support. We establish that this language-only ranking induces an objective mismatch, where language probability mass acts as a misspecified proxy for the intended multimodal task. Consequently, we reinterpret hallucination as a localized optimization error, a phenomenon where the decoder exploits language shortcuts to maximize a proxy score at the expense of visual grounding. To address this objective mismatch, we introduce VISAGE, a training-free decoding framework that calibrates the objective at inference time. VISAGE estimates the proxy discrepancy by quantifying the spatial entropy of cross-attention distributions. By enforcing a localization consensus across attention heads, the method penalizes spatially uniform distributions and re-ranks token commitments to favor visually grounded outcomes. We provide an analytical stability guarantee establishing that VISAGE maintains a bounded objective loss under estimation error. Evaluations across hallucination-sensitive and general-purpose benchmarks demonstrate the robustness of the framework, yielding relative gains of 8.59% on MMMU-val and 7.75% on HallusionBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。