调整训练顺序缓解多模态数据不平衡问题
Balance-aware Sequence Sampling Makes Multi-modal Learning Better
- 根据样本平衡度动态排序，分阶段引入训练数据
- 在多个数据集上优于现有最佳方法，提升模型鲁棒性
- 适合处理数据分布不均的多模态学习任务
为解决由数据异质性引发的模态不平衡问题，现有多模态学习（MML）方法主要从优化目标角度进行平衡。然而，几乎所有方法都忽略了样本序列的影响——不恰当的训练顺序会引发模型学习偏差，进一步加剧模态不平衡。本文提出平衡感知序列采样（BSS），以增强MML的鲁棒性。具体而言，我们首先定义一个多视角评估器，用于衡量每个样本的平衡程度；基于此，采用基于课程学习（CL）的启发式调度器，逐步提供训练子集，从平衡样本到不平衡样本递进训练，实现对MML的再平衡。此外，考虑到样本平衡度随模型能力提升而动态变化，我们设计了一种基于学习的概率采样方法，在每轮训练中动态更新训练序列，进一步提升性能。在多个常用数据集上的大量实验表明，该方法显著优于当前最优的MML方法。
原文摘要 · Abstract (English)
To address the modality imbalance caused by data heterogeneity, existing multi-modal learning (MML) approaches primarily focus on balancing this difference from the perspective of optimization objectives. However, almost all existing methods ignore the impact of sample sequences, i.e., an inappropriate training order tends to trigger learning bias in the model, further exacerbating modality imbalance. In this paper, we propose Balance-aware Sequence Sampling (BSS) to enhance the robustness of MML. Specifically, we first define a multi-perspective measurer to evaluate the balance degree of each sample. Via the evaluation, we employ a heuristic scheduler based on curriculum learning (CL) that incrementally provides training subsets, progressing from balanced to imbalanced samples to rebalance MML. Moreover, considering that sample balance may evolve as the model capability increases, we propose a learning-based probabilistic sampling method to dynamically update the training sequences at the epoch level, further improving MML performance. Extensive experiments on widely used datasets demonstrate the superiority of our method compared with state-of-the-art (SOTA) MML approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。