arXiv:2509.07864cs.CV2025-09被引 3

通过动态定位注意力错误,有效减少多模态大模型的幻觉问题。

Tracing and Mitigating Hallucinations in Multimodal LLMs via Dynamic Attention Localization

  • 基于层间注意力熵与头级聚焦度,精准定位幻觉源头。
  • 在图像描述和视觉问答任务上,幻觉率降低53%,准确率提升4%。
  • 无需额外训练,推理开销极低,适合实际部署场景。

多模态大语言模型在图像描述和视觉问答等任务中表现优异,但仍易产生与视觉输入矛盾的幻觉。现有方法虽尝试通过注意力机制检测并缓解幻觉,但通常对各层和注意力头采用统一修正策略,难以精确定位错误来源。本文首先揭示这些方法无法准确识别异常层。随后提出两种诊断工具:层图像注意力熵(LIAE)用于标记异常层,图像注意力聚焦度(IAF)用于评估层内注意力头的重要性。分析表明,LIAE能有效定位故障层,IAF可可靠排序需修正的注意力头。基于此,我们提出动态分层熵与注意力融合(D-LEAF),一种任务无关、注意力引导的推理阶段纠错方法,能动态定位并修正错误,计算开销极小。进一步地,我们建立D-LEAF与直接偏好优化(DPO)的理论关联,证明其有效性。实验显示,D-LEAF在标准图像描述基准上实现53%的相对性能提升,在视觉问答任务上准确率与F1分数均提高约4%,显著抑制幻觉且保持高效。

原文摘要 · Abstract (English)

Multimodal Large Language Models (MLLMs) achieve strong performance on tasks like image captioning and visual question answering, but remain prone to hallucinations, where generated text conflicts with the visual input. Prior work links this partly to insufficient visual attention, but existing attention-based detectors and mitigation typically apply uniform adjustments across layers and heads, obscuring where errors originate. In this paper, we first show these methods fail to accurately localize problematic layers. Then, we introduce two diagnostics: Layer Image Attention Entropy (LIAE) which flags anomalous layers, and Image Attention Focus (IAF) which scores attention heads within those layers. Analysis shows that LIAE pinpoints faulty layers and IAF reliably ranks heads that warrant correction. Guided by these signals, we propose Dynamic Layer-wise Entropy and Attention Fusion (D-LEAF), a task-agnostic, attention-guided method that dynamically localizes and corrects errors during inference with negligible overhead. Furthermore, by establishing a connection between D-LEAF and DPO, we provide theoretical justification for the effectiveness of D-LEAF. Results show our D-LEAF delivers a 53\% relative improvement on standard captioning benchmarks, and on VQA both accuracy and F1-score improve by approximately 4\%, substantially suppressing hallucinations while preserving efficiency.

多模态幻觉抑制注意力机制推理优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。