通过反事实推理与动态专家路由，减少多模态大模型的表面关联偏差。
Beyond Spurious Signals: Debiasing Multimodal Large Language Models via Counterfactual Inference and Adaptive Expert Routing
- 用反事实样本区分真实语义与虚假上下文，实现训练阶段去偏。
- 在多模态讽刺检测与情感分析任务中，性能超越现有最优模型。
- 适合研究多模态模型公平性、鲁棒性的研究人员参考。
多模态大语言模型（MLLMs）虽在融合视觉与文本信息方面表现出色，但常依赖表面相关性，影响其在复杂多模态推理任务中的鲁棒性与泛化能力。本文提出一种基于因果中介的去偏框架，通过反事实示例区分核心语义与虚假文本/视觉上下文，激活训练阶段的去偏机制，并采用带有动态路由的专家混合（MoE）架构，选择性地调用特定模态的去偏专家。在多模态讽刺检测与情感分析任务上的实证评估表明，该框架显著优于单模态去偏策略及现有最先进模型。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) have shown substantial capabilities in integrating visual and textual information, yet frequently rely on spurious correlations, undermining their robustness and generalization in complex multimodal reasoning tasks. This paper addresses the critical challenge of superficial correlation bias in MLLMs through a novel causal mediation-based debiasing framework. Specially, we distinguishing core semantics from spurious textual and visual contexts via counterfactual examples to activate training-stage debiasing and employ a Mixture-of-Experts (MoE) architecture with dynamic routing to selectively engages modality-specific debiasing experts. Empirical evaluation on multimodal sarcasm detection and sentiment analysis tasks demonstrates that our framework significantly surpasses unimodal debiasing strategies and existing state-of-the-art models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。