预测训练数据回收曲线，实现大模型高效数据教学顺序
Predicting Training Re-evaluation Curves Enables Effective Data Curriculums for LLMs
- 用最终模型回看训练批次，构建数据留存度曲线
- 提前预测曲线可优化数据排序，提升3.9B模型性能
- 适合大模型持续预训练与数据策略优化场景
数据课程已成为大模型成功训练的核心，但最优数据安排原则仍不明确。本文提出训练再评估曲线（TREC），一种利用最终模型权重回溯评估训练批次数据保留程度的诊断工具。TREC刻画了模型对数据的记忆能力随训练时序的变化规律。分析111M至3.9B参数模型的TREC发现，在TREC低谷期插入高质量数据能显著提升性能。重要的是，尽管TREC通常需训练后才能获得，我们证明其可基于AdamW的隐式指数移动平均系数提前预测，从而实现主动课程设计。通过预测已有训练方案的TREC，我们解释了以往消融实验结果，并揭示了次优数据布局。此外，我们将高质量数据对齐到TREC最小值处，用于优化一个3.9B参数、900B tokens持续预训练的LLM。
原文摘要 · Abstract (English)
Data curriculums have become central to successful LLM training, yet principles governing optimal data placement remain unclear. We introduce the *training re-evaluation curve (TREC)*, a diagnostic that retrospectively evaluates training batches *using the final model weights*. The TREC characterizes how well a trained model retains training data as a function of *when* the data was encountered during training. Analyzing TRECs for models from 111M to 3.9B parameters, we show that placing high-quality data at low points on the TREC significantly improves performance. Importantly, while a TREC is initially observable only after training, we demonstrate it can be *predicted in advance* from AdamW's implicit EMA coefficients, enabling proactive curriculum design. By predicting TRECs for published training recipes, we explain prior ablations and reveal suboptimal data placements. We also align high-quality data with TREC minima in order to improve continual pre-training of a 3.9B-parameter LLM trained on 900B tokens.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。