通过因果推理缓解多模态模型的幻觉问题，提升图文对齐准确性。
Mitigating Modality Prior-Induced Hallucinations in Multimodal Large Language Models via Deciphering Attention Causality
- 构建因果模型，将模态先验视为影响注意力与输出的混杂因子。
- 在视觉与语言注意力层应用反事实推理，使输出偏差降低65.3%。
- 无需重构模型，可直接插入现有系统，适合工业级部署。
多模态大语言模型（MLLMs）在产业和学术界备受关注，但常受视觉与语言先验偏见影响，导致多模态幻觉。这些偏见源于视觉编码器与大语言模型主干，影响负责对齐多模态输入的注意力机制。现有基于解码的方法仅关注统计相关性，忽视注意力机制与输出之间的因果关系，难以有效缓解此类偏见。为此，我们提出因果推断框架CausalMM，采用结构化因果建模，将模态先验视为注意力机制与输出之间的混杂因子。具体地，在视觉与语言注意力层级分别应用后门调整与反事实推理，减轻模态先验的负面影响，增强输入输出对齐。实验表明，该方法在6个VLind-Bench指标上最高提升65.3%，在MME Benchmark上提升164分，显著优于传统方法。大量实验证明其有效性且为即插即用方案。代码已开源：https://github.com/The-Martyr/CausalMM。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) have emerged as a central focus in both industry and academia, but often suffer from biases introduced by visual and language priors, which can lead to multimodal hallucination. These biases arise from the visual encoder and the Large Language Model (LLM) backbone, affecting the attention mechanism responsible for aligning multimodal inputs. Existing decoding-based mitigation methods focus on statistical correlations and overlook the causal relationships between attention mechanisms and model output, limiting their effectiveness in addressing these biases. To tackle this issue, we propose a causal inference framework termed CausalMM that applies structural causal modeling to MLLMs, treating modality priors as a confounder between attention mechanisms and output. Specifically, by employing backdoor adjustment and counterfactual reasoning at both the visual and language attention levels, our method mitigates the negative effects of modality priors and enhances the alignment of MLLM's inputs and outputs, with a maximum score improvement of 65.3% on 6 VLind-Bench indicators and 164 points on MME Benchmark compared to conventional methods. Extensive experiments validate the effectiveness of our approach while being a plug-and-play solution. Our code is available at: https://github.com/The-Martyr/CausalMM
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。