通过引导梯度方向提升多模态模型的均衡学习能力
Classifier-guided Gradient Modulation for Enhanced Multimodal Learning
- 基于分类器反馈调节各模态梯度的大小与方向
- 在四个数据集上均超越现有方法，显著提升性能
- 适合需要多模态融合且避免模态偏倚的研究场景
近年来多模态学习发展迅速，但在训练过程中，模型往往倾向于依赖学习速度较快的单一模态，导致其他模态利用不足。现有平衡方法在损失函数、优化器或模态数量上存在局限，且仅调节梯度幅度而忽略方向。为此，本文提出分类器引导的梯度调制（CGGM）方法，同时考虑梯度的大小与方向以实现更均衡的多模态学习。我们在 UPMC-Food 101、CMU-MOSI、IEMOCAP 和 BraTS 2021 四个数据集上进行了广泛实验，涵盖分类、回归和分割任务。结果表明，CGGM 在所有基线和先进方法中表现一致更优，验证了其有效性和通用性。代码已开源：https://github.com/zrguo/CGGM。
原文摘要 · Abstract (English)
Multimodal learning has developed very fast in recent years. However, during the multimodal training process, the model tends to rely on only one modality based on which it could learn faster, thus leading to inadequate use of other modalities. Existing methods to balance the training process always have some limitations on the loss functions, optimizers and the number of modalities and only consider modulating the magnitude of the gradients while ignoring the directions of the gradients. To solve these problems, in this paper, we present a novel method to balance multimodal learning with Classifier-Guided Gradient Modulation (CGGM), considering both the magnitude and directions of the gradients. We conduct extensive experiments on four multimodal datasets: UPMC-Food 101, CMU-MOSI, IEMOCAP and BraTS 2021, covering classification, regression and segmentation tasks. The results show that CGGM outperforms all the baselines and other state-of-the-art methods consistently, demonstrating its effectiveness and versatility. Our code is available at https://github.com/zrguo/CGGM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。