arXiv:2606.29858cs.CL2026-06

发现语言模型训练中损失下降的规律,源于令牌学习时间的分布差异。

Smooth Scaling Laws Hide Stepwise Token Learning

论文配图:Smooth Scaling Laws Hide Stepwise Token Learning
图 1 · 摘自论文原文
  • 将损失变化分解为单个令牌的学习过程,发现学习集中在局部跃迁阶段。
  • 在100多次训练实验中,验证了学习时间谱可精确复现损失导数变化趋势。
  • 调整训练数据分布以匹配学习时机,使验证损失下降速度提升11%。

语言模型的损失遵循模型规模和数据规模上的规律性缩放定律,但其背后的机制尚不明确。现有解释常归因于自然语言中模式难度的重尾分布,但这一观点尚未在大规模真实数据训练中以令牌级粒度得到直接验证。本文提出一种令牌级分析框架,将缩放定律分解为单个上下文化令牌的局部学习事件。通过用sigmoid函数拟合令牌损失轨迹,我们发现令牌学习集中于局部跃迁阶段,形成主导缩放定律形态的学习时间谱。在使用现代大语言模型架构、覆盖超过一百次预训练运行、模型规模达60亿参数、训练数据达3000亿令牌的大规模真实语料上,测量到的学习时间谱能定量重建验证损失在训练步数 $T$、数据规模 $D$、模型规模 $M$ 三个维度上的导数。进一步表明该信号具有实际应用价值:通过根据令牌可学习时间重新塑造训练分布,可改变优化轨迹,实现验证损失减少速度提升11%。这些结果提供了直接实证证据,表明缩放定律主要由令牌级学习时间分布决定,且该分布不仅可用于解释训练行为,还可用于优化训练性能。

原文摘要 · Abstract (English)

Language model loss follows remarkably regular scaling laws over model and data size, yet it remains unclear why the aggregate loss should exhibit a power-law form. Existing explanations often attribute this regularity to a heavy-tailed spectrum of pattern difficulty in natural language, but this view has not been directly validated at token-level granularity in large-scale real-data training. We present a token-level framework that decomposes scaling laws into localized learning events of individual contextualized tokens. By fitting token loss trajectories with sigmoids, we show that token learning is concentrated in localized transitions, giving rise to a learning-time spectrum that dominates the scaling-law shape. Across more than one hundred pre-training runs on large and diverse real-language corpora with modern LLM architectures, scaling up to 6B parameters and 300B training tokens, the measured learning-time spectrum quantitatively reconstructs the validation loss derivative along the training-step $T$, data-scale $D$, and model-scale $M$ axes. We further show that the same signal is actionable: by reshaping the training distribution according to when tokens become learnable, we alter the optimization trajectory and achieve 11\% faster validation-loss reduction. These results provide direct empirical evidence that scaling laws are governed primarily by the distribution of token-level learning times, and that this distribution can be used not only to explain scaling behavior but also to improve training performance.

缩放定律训练优化大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。