通过调控注意力动态,显著降低多模态模型幻觉率。
ADAPT: Attention Dynamics Alignment with Preference Tuning for Faithful MLLMs

- 直接干预文本到图像的交叉注意力演化过程
- 在主流模型上降低40%-60%幻觉率,保持多模态能力
- 适合关注模型可解释性与生成真实性的研究者
多模态大语言模型(MLLMs)严重受限于幻觉问题,即生成内容与给定图像不一致。本文发现幻觉的内部信号是生成过程中文本到图像交叉注意力的逐步退化,导致注意力分散或偏移等典型失败模式。现有方法多为结果导向,未针对此内在机制。为此,我们提出ADAPT(注意力动态对齐与偏好微调),一个基于注意力的框架:利用早期解码中提取的视觉锚点提供稳定空间定位;设计注意力监督推理机制,实时检测并纠正注意力漂移;引入视觉注意力引导DPO,对齐偏好以生成视觉一致的回答。实验表明,每个组件均有效减少幻觉,完整框架在多个主流骨干模型上实现新最优表现,幻觉率下降40%-60%,同时保持通用多模态能力。本工作从模型内部注意力行为出发,提供了抑制幻觉的新视角。代码已公开于https://github.com/yao-ustc/ADAPT。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) are critically hampered by hallucination, generating content inconsistent with the provided image. In this paper, we identify an internal signature of hallucination: progressive degradation of text-to-image cross-attention during generation, leading to specific failure patterns like unfocused or biased attention. Existing mitigation strategies are largely outcome-driven and do not explicitly target this failure mode. To address this problem, we propose ADAPT (Attention Dynamics Alignment with Preference Tuning), an attention-based framework that intervenes directly on text-to-image cross-attention dynamics. We propose ADAPT with three key contributions: a cross-attention visual anchor refined from early decoding to provide stable spatial grounding, an attention-supervised inference mechanism that detects and corrects attention drift online, and a Visual Attention Guidance DPO that aligns preferences toward visually grounded responses. Experiments show that each component of ADAPT contributes to hallucination reduction, and the full framework achieves new best results across multiple hallucination benchmarks, reducing hallucination rates by 40%-60% across mainstream backbones while preserving general multimodal capabilities. Our work provides an attention-based perspective on mitigating hallucinations by exploring the model's internal text-to-image cross-attention behaviors. Code is available at https://github.com/yao-ustc/ADAPT
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。