利用训练中保存的模型检查点自动混合数据，提升语言模型性能。
AutoMixer: Checkpoint Artifacts as Automatic Data Mixers
- 用检查点模型在训练过程中的能力表现，自动选择并混合数据。
- 在8个推理基准上实现最高1.93%的性能提升。
- 适合关注预训练数据优化与高效训练策略的研究者。
在语言模型训练中，希望模型具备多种任务能力，但难以直接获得合适的训练数据组合，因数据与任务间的关系难建模。本文发现，训练轨迹中不同时间点的检查点模型会涌现出不同的能力。这些检查点作为训练过程中的未充分利用的副产品，可基于其在基准测试上的表现被识别，并通过它们对源数据的一阶影响近似聚合，作为数据混合器使用。在8个推理基准上的实验表明，该框架在预训练设置下显著提升性能，最高达1.93%。结果展示了检查点模型在提升数据质量与优化数据混合方面的潜力。
原文摘要 · Abstract (English)
In language model training, it is desirable to equip models with capabilities from various tasks. However, it is not clear how to directly obtain the right data mixtures for these capabilities as the relationship between data and tasks is difficult to be modeled. In this work, we observe that checkpoint models exhibit emerging capabilities at different points in the training trajectory. Often, the training process saves checkpoints as artifacts that are under-utilized as a source of in-training data signals. We identify these artifact models based on their respective capabilities on the benchmarks and leverage them as data mixers by using their aggregated first-order influence approximation over source data. We demonstrated on eight reasoning benchmarks that the proposed framework shows significant improvements in the pretraining setting, with performance improvements of up to 1.93%. Overall, this shows the potential of checkpoint models to enhance data quality and optimize data mixtures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。