解决多模态大模型因无关模态干扰导致的判断失准问题
Diagnosing and Mitigating Modality Interference in Multimodal Large Language Models
- 通过因果与扰动实验系统诊断模态干扰来源
- 新方法在多个基准上提升单模态鲁棒性与泛化能力
- 适合关注多模态模型可靠性与可解释性的研究者
多模态大语言模型在多模态基准上表现强劲，但在遭遇无关模态干扰（如视觉理解中的无关文本或问答中的无关视觉内容）时往往表现出较差的鲁棒性。模态干扰的核心问题是非必要模态中的虚假信号扭曲模型决策。我们通过因果分析与基于扰动的诊断实验系统地探究该问题。为应对此挑战，提出统一微调框架：结合启发式与对抗性扰动的数据增强，以及原始输入与扰动输入间的输出一致性正则化。在图像主导、文本主导及多模态基准上的大量实验表明，该方法在多种MLLM架构和模型规模下，均一致提升了单模态鲁棒性与泛化性能，同时保持标准多模态任务表现。
原文摘要 · Abstract (English)
Multimodal Large Language Models demonstrate strong performance on multimodal benchmarks, yet often exhibit poor robustness when exposed to spurious modality interference, such as irrelevant text in vision understanding, or irrelevant visual content in question answering. At its core, modality interference refers to cases where spurious signals from non-essential modalities distort model decisions, which we systematically analyze through causal, perturbation-based diagnostic experiments. To address this problem, we propose a unified finetuning framework that combines heuristic and adversarial perturbation-based data augmentation with output-level consistency regularization between original and perturbed inputs. Extensive experiments across image-heavy, text-heavy, and multimodal benchmarks, spanning multiple MLLM architectures and model scales, demonstrate consistent improvements in unimodal robustness and generalization, while improving standard multimodal performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。