解释大模型训练慢的根源：损失下降呈1/3次幂规律
Universal One-third Time Scaling in Learning Peaked Distributions
- softmax与交叉熵导致峰值分布学习时损失缓慢下降
- 损失随时间呈1/3次幂衰减，与模型细节无关
- 揭示训练效率瓶颈，为加速训练提供新思路
大语言模型训练成本高昂，部分原因在于损失呈现缓慢的幂律收敛，其根源尚不明确。通过对简化模型的系统分析和对大语言模型的实证研究，我们发现这种行为可由softmax与交叉熵的使用内在引发。在学习峰值概率分布（如下一个词分布）时，这些组件会普遍导致损失和梯度的幂律衰减，与诸多微观细节无关，形成根本性优化瓶颈。这最终导致损失随时间呈现幂律缩放，且具有普适的指数1/3。该结果为观测到的神经网络缩放现象提供了机制解释，并指出了提升大模型训练效率的新方向。
原文摘要 · Abstract (English)
Training large language models (LLMs) is computationally expensive, partly because the loss exhibits slow power-law convergence whose origin remains debatable. Through systematic analysis of toy models and empirical evaluation of LLMs, we show that this behavior can arise intrinsically from the use of softmax and cross-entropy. When learning peaked probability distributions, e.g., next-token distributions, these components generically yield power-law vanishing losses and gradients, regardless of many microscopic details, creating a fundamental optimization bottleneck. This ultimately leads to power-law time scaling of the loss with a universal exponent of $1/3$. Our results provide a mechanistic explanation for observed neural scaling and suggest new directions for improving LLM training efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。