用因果解耦减少多模态大模型的幻觉生成
Causal-LLaVA: Causal Disentanglement for Mitigating Hallucination in Multimodal Large Language Models
- 通过因果干预分离视觉与语言中的虚假关联
- 在多个基准测试中显著降低幻觉率，保持强性能
- 适合关注模型可信度与公平性的研究者
多模态大语言模型（MLLMs）在视觉理解任务中表现优异，但常出现物体幻觉——生成输入中不存在或不一致的物体描述。这与数据集偏差密切相关，即物体频繁共现导致跨模态语义表征纠缠。因此，模型可能错误激活与输入相关但实际未出现的物体表征。为此，我们提出一种基于因果驱动的解耦框架，通过因果干预缓解幻觉问题。方法包括视觉路径中的因果驱动投影器和语言模型最后一层的因果干预模块，共同减少训练数据偏差引发的虚假相关性。实验结果表明，该方法显著降低幻觉率，同时在多个多模态基准上保持优异性能。可视化分析进一步验证了物体表征的更好可分性。代码已开源：https://github.com/IgniSavium/Causal-LLaVA
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) have demonstrated strong performance in visual understanding tasks, yet they often suffer from object hallucinations--generating descriptions of objects that are inconsistent with or entirely absent from the input. This issue is closely related to dataset biases, where frequent co-occurrences of objects lead to entangled semantic representations across modalities. As a result, models may erroneously activate object representations that are commonly associated with the input but not actually present. To address this, we propose a causality-driven disentanglement framework that mitigates hallucinations through causal intervention. Our approach includes a Causal-Driven Projector in the visual pathway and a Causal Intervention Module integrated into the final transformer layer of the language model. These components work together to reduce spurious correlations caused by biased training data. Experimental results show that our method significantly reduces hallucinations while maintaining strong performance on multiple multimodal benchmarks. Visualization analyses further confirm improved separability of object representations. The code is available at: https://github.com/IgniSavium/Causal-LLaVA
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。