动态调整学习难度,让多模态模型更均衡高效
DynCIM: Dynamic Curriculum for Imbalanced Multimodal Learning
- 按样本难易度和模态贡献动态调整学习顺序
- 在6个数据集上超越现有方法,提升模型鲁棒性
- 适合处理数据质量不均的多模态任务
多模态学习通过融合不同模态的互补信息提升决策能力,但因数据质量差异和模态表征能力不平衡,其潜力尚未充分释放。为此,我们提出DynCIM,一种新型动态课程学习框架,从样本和模态两个层面量化内在不平衡。该框架在样本层面基于预测偏差、一致性与稳定性动态评估每个样本的学习难度,在模态层面从全局与局部角度衡量各模态贡献。此外,引入基于门控的动态融合机制,自适应调节模态权重,减少冗余并优化融合效果。在涵盖双模态与三模态场景的六个基准数据集上的大量实验表明,DynCIM持续优于当前最优方法。该方法有效缓解了模态与样本层面的不平衡,提升了多模态学习任务中的适应性与鲁棒性。代码已公开于https://github.com/Raymond-Qiancx/DynCIM。
原文摘要 · Abstract (English)
Multimodal learning integrates complementary information from diverse modalities to enhance the decision-making process. However, the potential of multimodal collaboration remains under-exploited due to disparities in data quality and modality representation capabilities. To address this, we introduce DynCIM, a novel dynamic curriculum learning framework designed to quantify the inherent imbalances from both sample and modality perspectives. DynCIM employs a sample-level curriculum to dynamically assess each sample's difficulty according to prediction deviation, consistency, and stability, while a modality-level curriculum measures modality contributions from global and local. Furthermore, a gating-based dynamic fusion mechanism is introduced to adaptively adjust modality contributions, minimizing redundancy and optimizing fusion effectiveness. Extensive experiments on six multimodal benchmarking datasets, spanning both bimodal and trimodal scenarios, demonstrate that DynCIM consistently outperforms state-of-the-art methods. Our approach effectively mitigates modality and sample imbalances while enhancing adaptability and robustness in multimodal learning tasks. Our code is available at https://github.com/Raymond-Qiancx/DynCIM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。