arXiv:2505.20862cs.CV2025-05NeurIPS被引 21

提出AVCD框架,用动态注意力抑制音视频语言模型的幻觉。

AVCD: Mitigating Hallucinations in Audio-Visual Large Language Models through Contrastive Decoding

  • 根据注意力分布动态识别弱模态,用掩码生成干扰输出
  • 在AVHBench上使VideoLLaMA2准确率提升2%,video-SALMONN提升7%
  • 无需训练,适配多模态交互,适合音视频理解任务

幻觉仍是多模态大语言模型(MLLMs)的主要挑战。尽管对比解码(CD)在视觉-语言模型中展现潜力,但其难以应对音视频-语言模型(AV-LLMs)中由单模态或跨模态组合引发的幻觉。为此,本文提出音频-视觉对比解码(AVCD)——一种无需训练的解码框架,可建模三模态交互并抑制模态诱发的幻觉。不同于以往仅扰动单一模态的CD方法,AVCD利用注意力分布动态识别次要模态,并通过感知掩码生成扰动输出概率。同时,重构原有CD框架以支持音频、视频和文本三输入联合处理。为提升效率,引入基于熵的自适应解码策略,依据模型置信度跳过冗余解码步骤。大量实验表明,AVCD持续优于现有方法:在AVHBench数据集上,对VideoLLaMA2准确率提升2%,对video-SALMONN提升7%,展现出强鲁棒性与泛化能力。代码已开源。

原文摘要 · Abstract (English)

Hallucination remains a major challenge in multimodal large language models (MLLMs). To address this, various contrastive decoding (CD) methods have been proposed that contrasts original logits with hallucinated logits generated from perturbed inputs. While CD has shown promise in vision-language models (VLMs), it is not well-suited for AV-LLMs, where hallucinations often emerge from both unimodal and cross-modal combinations involving audio, video, and language. These intricate interactions call for a more adaptive and modality-aware decoding strategy. In this paper, we propose Audio-Visual Contrastive Decoding (AVCD)-a novel, training-free decoding framework designed to model trimodal interactions and suppress modality-induced hallucinations in AV-LLMs. Unlike previous CD methods in VLMs that corrupt a fixed modality, AVCD leverages attention distributions to dynamically identify less dominant modalities and applies attentive masking to generate perturbed output logits. To support CD in a trimodal setting, we also reformulate the original CD framework to jointly handle audio, visual, and textual inputs. Finally, to improve efficiency, we introduce entropy-guided adaptive decoding, which selectively skips unnecessary decoding steps based on the model's confidence in its predictions. Extensive experiments demonstrate that AVCD consistently outperforms existing decoding methods. Especially, on the AVHBench dataset, it improves accuracy by 2% for VideoLLaMA2 and 7% for video-SALMONN, demonstrating strong robustness and generalizability. Our code is available at https://github.com/kaistmm/AVCD.

多模态幻觉抑制解码优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。