动态调整数据混合权重，提升大规模多任务训练效率
PiKE: Adaptive Data Mixing for Large-Scale Multi-Task Learning Under Low Gradient Conflicts
- 根据梯度正向交互自适应调节数据采样权重
- 在无显著梯度冲突场景下实现更快收敛和更好性能
- 适合大规模多语言/多领域预训练，计算开销极低
现代基础模型通过多样化数据集训练以提升跨任务与跨域泛化能力。其核心挑战在于如何有效混合与采样多源数据，这自然引出多任务学习（MTL）视角。以往工作聚焦缓解梯度冲突，但我们观察到大规模预训练（如多语言或跨领域训练）中梯度冲突通常极小甚至不存在。基于此，我们提出PiKE（基于正向梯度交互的K任务权重估计器），一种动态调整训练中采样权重的自适应数据混合算法。PiKE利用非冲突梯度交互，最小化每步平均损失下降的近紧上界，同时计算开销可忽略。我们提供理论收敛保证，并证明PiKE优于静态及非自适应混合基线。进一步，我们将PiKE扩展以促进任务间平衡学习。大规模语言模型预训练实验表明，相比现有方法，PiKE实现更快收敛与更优下游性能。
原文摘要 · Abstract (English)
Modern foundation models are trained on diverse datasets to enhance generalization across tasks and domains A central challenge in this process is determining how to effectively mix and sample data from multiple sources This naturally leads to a multitask learning (MTL) perspective While prior work in MTL has emphasized mitigating gradient conflicts we observe that largescale pretraining scenariossuch as multilingual or multidomain trainingoften exhibit little to no gradient conflict Motivated by this observation we propose PiKE (Positive gradient interaction-based K-task weights Estimator) an adaptive data mixing algorithm that dynamically adjusts sampling weights during training PiKE exploits nonconflicting gradient interactions to minimize a neartight upper bound on the average loss decrease at each step while incurring negligible computational overhead We provide theoretical convergence guarantees and show that PiKE outperforms static and nonadaptive mixing baselines Furthermore we extend PiKE to promote balanced learning across tasks Extensive experiments on largescale language model pretraining confirm that PiKE achieves faster convergence and improved downstream performance compared to existing approaches
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。