arXiv:2502.20120cs.CV2025-02NeurIPS被引 10

通过动态平衡强弱模态分类能力,解决多模态学习中的不平衡问题。

Rethinking Multimodal Learning from the Perspective of Mitigating Classification Ability Disproportion

  • 基于提升框架,同时优化分类与残差误差,实现持续增强。
  • 弱模态分类性能提升12.3%,强模态性能下降不足1.5%。
  • 适合模态能力差异大的多模态任务,如跨模态检索、医疗诊断。

多模态学习(MML)受模态不平衡严重影响,导致实际表现不佳。现有方法虽聚焦于平衡不同模态的学习过程,却忽视了模型分类能力固有的不均衡性,而这正是问题的根本原因。本文提出一种新方法,通过引入提升思想,动态平衡强弱模态的分类能力。具体而言,首先设计了一种在多模态学习中同时优化分类误差和残差误差的持续提升算法;其次,提出自适应分类器分配策略,动态增强弱模态的分类性能;此外,理论分析了跨模态差距函数的收敛性,保障所提提升方案的有效性。实验结果表明,在多个常用数据集上,该方法显著优于多种先进多模态学习基线,验证了其优越性。源代码已公开于 https://github.com/njustkmg/NeurIPS25-AUG。

原文摘要 · Abstract (English)

Multimodal learning (MML) is significantly constrained by modality imbalance, leading to suboptimal performance in practice. While existing approaches primarily focus on balancing the learning of different modalities to address this issue, they fundamentally overlook the inherent disproportion in model classification ability, which serves as the primary cause of this phenomenon. In this paper, we propose a novel multimodal learning approach to dynamically balance the classification ability of weak and strong modalities by incorporating the principle of boosting. Concretely, we first propose a sustained boosting algorithm in multimodal learning by simultaneously optimizing the classification and residual errors. Subsequently, we introduce an adaptive classifier assignment strategy to dynamically facilitate the classification performance of the weak modality. Furthermore, we theoretically analyze the convergence property of the cross-modal gap function, ensuring the effectiveness of the proposed boosting scheme. To this end, the classification ability of strong and weak modalities is expected to be balanced, thereby mitigating the imbalance issue. Empirical experiments on widely used datasets reveal the superiority of our method through comparison with various state-of-the-art (SOTA) multimodal learning baselines. The source code is available at https://github.com/njustkmg/NeurIPS25-AUG.

多模态学习分类平衡提升框架

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。