通过自评估调节模态权重，有效减少多模态模型的跨模态幻觉。
MAD: Modality-Adaptive Decoding for Mitigating Cross-Modal Hallucinations in Multimodal Large Language Models
- 基于任务需求自适应调整各模态解码分支权重。
- 在CMM和AVHBench上使幻觉率降低7.8%~8.7%。
- 无需训练，适用于多种音频-视觉语言模型。
多模态大语言模型（MLLMs）存在跨模态幻觉问题，即一种模态错误影响对另一种模态的生成，导致虚构内容。这暴露了模态交互控制的根本缺陷。为此，我们提出训练无关的模态自适应解码（MAD），根据任务需求动态调整模态特定解码分支的权重。MAD通过查询模型自身判断每项任务所需模态，提取模态概率以自适应加权对比解码分支，使模型聚焦相关信息并抑制跨模态干扰。在CMM和AVHBench上的实验表明，MAD显著降低跨模态幻觉，对VideoLLaMA2-AV提升7.8%和2.0%，对Qwen2.5-Omni提升8.7%和4.7%。结果表明，通过自评估实现显式模态感知对鲁棒多模态推理至关重要，为现有对比解码方法提供原则性扩展。代码已开源。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) suffer from cross-modal hallucinations, where one modality inappropriately influences generation about another, leading to fabricated output. This exposes a more fundamental deficiency in modality-interaction control. To address this, we propose Modality-Adaptive Decoding (MAD), a training-free method that adaptively weights modality-specific decoding branches based on task requirements. MAD leverages the model's inherent ability to self-assess modality relevance by querying which modalities are needed for each task. The extracted modality probabilities are then used to adaptively weight contrastive decoding branches, enabling the model to focus on relevant information while suppressing cross-modal interference. Extensive experiments on CMM and AVHBench demonstrate that MAD significantly reduces cross-modal hallucinations across multiple audio-visual language models (7.8\% and 2.0\% improvements for VideoLLaMA2-AV, 8.7\% and 4.7\% improvements for Qwen2.5-Omni). Our approach demonstrates that explicit modality awareness through self-assessment is crucial for robust multimodal reasoning, offering a principled extension to existing contrastive decoding methods. Our code is available at \href{https://github.com/top-yun/MAD}{https://github.com/top-yun/MAD}
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。