通过动态调整弱模态训练优先级,提升多模态融合的平衡性与性能。
Modality Equilibrium Matters: Minor-Modality-Aware Adaptive Alternating for Cross-Modal Memory Enhancement
- 基于谢尔普利值自适应调度训练顺序,优先强化弱模态
- 在四个基准数据集上实现当前最优的平衡性与准确率
- 适用于模态缺失场景,适合多模态模型开发者参考
多模态融合易受模态不平衡影响,主导模态会压制弱模态,导致学习偏斜和融合效果不佳,尤其在模态不完整时更显著。为此,本文提出一种基于谢尔普利值的交替训练框架,自适应地优先处理弱模态以实现平衡并增强融合效果。该方法利用谢尔普利值指导训练序列调度,确保欠优化模态获得充分学习。同时引入记忆模块,通过跨模态映射机制在特征与样本层面对齐,精炼并继承模态特定表示。为验证方法的泛化能力,编码器模块采用传统与基于LLM的骨干网络。构建新的多模态平衡度量指标——平衡偏差度量(EDM),在四个多模态基准数据集上评估,本方法取得当前最优(SOTA)表现。缺失模态下的鲁棒性分析进一步证明其强泛化能力。研究揭示了交替训练的未被挖掘潜力,表明战略性模态优先可从根本上平衡并促进多模态学习,为优化多模态训练动态提供新范式。
原文摘要 · Abstract (English)
Multimodal fusion is susceptible to modality imbalance, where dominant modalities overshadow weak ones, easily leading to biased learning and suboptimal fusion, especially for incomplete modality conditions. To address this problem, we propose a Shapley-guided alternating training framework that adaptively prioritizes minor modalities to balance and thus enhance the fusion. Our method leverages Shapley Value-based scheduling to improve the training sequence adaptively, ensuring that under-optimized modalities receive sufficient learning. Additionally, we introduce the memory module to refine and inherit modality-specific representations with a cross-modal mapping mechanism to align features at both the feature and sample levels. To further validate the adaptability of the proposed approach, the encoder module empirically adopts both conventional and LLM-based backbones. With building up a novel multimodal equilibrium metric, namely, equilibrium deviation metric (EDM), we evaluate the performance in both balance and accuracy across four multimodal benchmark datasets, where our method achieves state-of-the-art (SOTA) results. Meanwhile, robustness analysis under missing modalities highlights its strong generalization capabilities. Accordingly, our findings reveal the untapped potential of alternating training, demonstrating that strategic modality prioritization fundamentally balances and promotes multimodal learning, offering a new paradigm for optimizing multimodal training dynamics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。