调整预训练数据顺序能提升小模型训练稳定性，但大模型效果趋同。
Curriculum Learning for LLM Pretraining: An Analysis of Learning Dynamics
- 用语言学启发的三种排序方式（年龄获取、词频、动词变化）对比随机顺序。
- 小模型在非随机排序下梯度噪声更低，输出头奇异值熵更平稳。
- 方向很重要：升序优于降序，尤其在14M-70M参数规模时效果显著。
课程学习通过改变预训练数据顺序来影响学习动态，但其机制尚不明确。本研究在3000亿个词元上对1400万至10亿参数的模型进行预训练，采用三种语言学启发的课程——获取年龄、词频、动词变化（VV），并与随机顺序对比。分析了潜在训练阶段、梯度噪声尺度（GNS）及输出头的奇异值结构。结果发现，所有模型经历相似的隐含阶段，课程主要影响各阶段持续时间。随机顺序在1400万至7000万参数时产生更高GNS，且在1.6亿参数前出现输出头奇异熵峰值，表明梯度更嘈杂、输出头过饱和。反向的VV对照实验显示，顺序方向至关重要：降序丢失了升序带来的精度优势。在更大规模下，这些稳定性差异减小。结论表明，所研究的课程主要提升小模型在各阶段内的训练稳定性，而非引入新阶段。
原文摘要 · Abstract (English)
Curriculum learning changes the order of pretraining data, but it remains unclear how ordering changes the learning dynamics. We pretrain models from 14M to 1B parameters for 300B tokens under three linguistically motivated curricula--Age-of-Acquisition, word frequency, and Verb Variation (VV)--and compare each against Random ordering. We analyze latent training phases, gradient noise scale (GNS), and the singular-value structure of the output head. We find that training follows a shared sequence of latent phases, while curricula mainly change time spent in each phase. Random ordering yields higher GNS at 14M-70M and late singular-entropy spikes up to 160M, consistent with noisier gradients and output-head saturation. A reverse-order VV control shows that direction matters: descending order loses much of the accuracy advantage of the ascending curriculum. At larger scales, these stability differences are smaller. These results indicate that the curricula studied here are associated with more stable within-phase training in smaller models rather than with the creation of new phases.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。