用梯度引导蒸馏提升弱模态表现，解决多模态模型失衡问题。
G$^{2}$D: Boosting Multimodal Learning with Gradient-Guided Distillation
- 基于梯度设计蒸馏损失，融合单模态与多模态目标。
- 动态轮换模态优先级，避免强模态压制弱模态。
- 在分类与回归任务中超越现有方法，尤其增强弱模态作用。
多模态学习旨在融合多种数据模态以获得更全面的性能。然而，传统多模态模型常面临模态失衡问题，即一个或少数几个模态主导优化过程，导致特征表示不佳且弱模态未被充分利用。为此，本文提出梯度引导蒸馏（G²D），一种知识蒸馏框架，通过自定义损失函数融合单模态与多模态目标来优化多模态模型。G²D进一步引入动态顺序模态优先机制（SMP），在训练过程中轮流让各模态主导学习，避免强模态压制弱模态。我们在多个真实世界数据集上验证了G²D，结果表明其能显著增强弱模态的作用，并在分类与回归任务中优于当前最优方法。代码已公开于https://github.com/rAIson-Lab/G2D。
原文摘要 · Abstract (English)
Multimodal learning aims to leverage information from diverse data modalities to achieve more comprehensive performance. However, conventional multimodal models often suffer from modality imbalance, where one or a few modalities dominate model optimization, leading to suboptimal feature representation and underutilization of weak modalities. To address this challenge, we introduce Gradient-Guided Distillation (G$^{2}$D), a knowledge distillation framework that optimizes the multimodal model with a custom-built loss function that fuses both unimodal and multimodal objectives. G$^{2}$D further incorporates a dynamic sequential modality prioritization (SMP) technique in the learning process to ensure each modality leads the learning process, avoiding the pitfall of stronger modalities overshadowing weaker ones. We validate G$^{2}$D on multiple real-world datasets and show that G$^{2}$D amplifies the significance of weak modalities while training and outperforms state-of-the-art methods in classification and regression tasks. Our code is available at https://github.com/rAIson-Lab/G2D.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。