通过李括号分析训练顺序对多领域模型的影响，发现特定参数区域能提升性能。
Commute Your Domains: Trajectory Optimality Criterion for Multi-Domain Learning
- 用梯度向量场的李括号研究训练顺序的微小影响
- 在参数空间中识别出改变顺序可优化目标损失的区域
- 验证于小规模模型和双语大模型预训练，适用于需优化数据混合的场景
在多领域学习中，一个模型在多个数据领域上进行训练，以利用共享知识并提升泛化能力。训练时不同领域数据的顺序（或数据混合方式）可能显著影响模型在各领域上的表现，但这一依赖关系尚未得到充分研究。本文基于梯度向量场的李括号概念，分析训练顺序变化的瞬时效应，识别出在参数空间中改变两个训练领域顺序能改善目标损失的区域。我们通过一个简单示例以及双语大语言模型预训练任务，验证了该理论框架对训练顺序（或数据混合）影响的预测有效性。
原文摘要 · Abstract (English)
In multi-domain learning, a single model is trained on diverse data domains to leverage shared knowledge and improve generalization. The order in which the data from these domains is used for training can significantly affect the model's performance on each domain. However, this dependence is under-studied. In this paper, we investigate the influence of training order (or data mixing) in multi-domain learning using the concept of Lie bracket of gradient vector fields. By analyzing the infinitesimal effects of changing the training order, we identify regions in the parameter space where altering the order between two training domains can benefit the target loss. We validate the predictions of our theoretical framework on the influence of training order (or data mixing) both on a toy example and bilingual LLM pre-training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。