通过重塑标签空间,解决多模态学习中模态训练不平衡问题
Balancing Multimodal Learning through Label Space Reshaping

- 从标签空间设计入手,均衡不同模态的特征映射难度
- 在多个模型架构上实现性能提升,跨模态交互更充分
- 适合关注多模态对齐与协同训练的研究者使用
多模态学习常面临模态不平衡问题:收敛快的模态主导优化过程,而其他模态则训练不足。现有方法通常通过增强弱模态或调整优化梯度来缓解,但主要针对优化速率差异,往往牺牲强模态的优化能力,且未深入分析模态层面差异的根源。基于理论分析与实证观察,我们指出学习速率差异源于模态特定特征空间到共享标签空间映射难度的不同。为此,提出首个从标签侧设计的平衡方法——平衡多模态标签重塑(BMLR)。BMLR通过重塑跨模态标签空间,使各模态的映射难度趋于一致,从而促进模态间交互,并向每个模态注入更丰富的类别间信息。在多种架构上的大量实验表明,BMLR持续提升多模态性能,且与不同模型设计具有强兼容性。源代码即将发布。
原文摘要 · Abstract (English)
Multimodal learning often suffers from modality imbalance, where modalities that converge faster dominate optimization while others remain undertrained. Existing approaches typically mitigate this issue by strengthening the weak modality or adjusting optimization gradients. However, such strategies mainly compensate for optimization rate discrepancies, often at the expense of the strong modality's optimization capacity, without analyzing how these discrepancies arise at the modality level. Based on theoretical insights and empirical observations, we argue that the discrepancy of learning pace arises from differences in the mapping difficulty between modality-specific feature space and the shared label space. To address this issue, we propose Balanced Multimodal Label Reshaping (BMLR), the first method that promotes multimodal balance from the label-side design. BMLR reshapes the cross-modal label space to equalize mapping difficulty across modalities, thereby facilitating modality interaction and injecting richer inter-class information into each modality. Extensive experiments across multiple architectures demonstrate that BMLR consistently improves multimodal performance and exhibits strong compatibility with diverse model designs. The source code will be released soon.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。