发现大模型训练学习率缩放存在非线性，提出用有效学习率改进预测精度。
On the Nonlinearity of Learning Rate Scaling for LLM Training
- 用有效学习率替代原始学习率，消除大模型缩放中的非线性偏差。
- 在50亿到1000亿词数据上训练2200万至7亿参数模型，发现学习率需向上弯曲调整。
- 适合需要高效训练大模型的实践者，尤其关注学习率调度与优化器设计的人。
学习率迁移可降低大语言模型训练成本：无需在目标规模上遍历学习率，只需从较小规模外推。现有方法常假设最优学习率随数据量和模型规模呈对数线性缩放。我们对2200万至7亿参数、在50亿至1000亿词数据上训练的GPT-2型模型进行了细致实证研究，发现最优学习率在大规模下出现向上弯曲，导致外推不准。我们发现，当以有效学习率（归一化权重空间中的步长）替代学习率，或改用数据量D外推而非模型规模N外推时，这种弯曲现象基本消失。进一步分析表明：当最优学习率较小时，权重范数收敛到平衡态更慢，需更大步长以缩短瞬态阶段。使用直接控制有效学习率的AdamH进行实验，进一步验证了该解释。
原文摘要 · Abstract (English)
Learning-rate transfer can reduce the cost of training large language models: instead of sweeping learning rates at target scale, practitioners extrapolate from smaller runs. Existing approaches often assume that the optimal learning rate follows a log-linear scaling law in data scale and model size. We carefully examine and evaluate this scaling law. In our empirical study of GPT-2--style models from 22M to 707M parameters trained on 5B to 100B tokens, the optimal learning rate develops upward curvature at larger scales, leading to inaccurate extrapolation. We find that this curvature largely disappears when learning rates are replaced by effective learning rate (the step size in normalized weight space), and when data $D$ extrapolation is used instead of model size $N$ extrapolation. Next, we explain nonlinearity in scaling: weight-norm converges to equilibrium slower when optimal learning is small, requiring a larger step size to reduce the transient phase. Experiments with AdamH, which directly controls the effective learning rate, further support this explanation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。