提出大模型训练中权重衰减与批量大小的缩放规律，可提前预测最优超参。
Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-training
- 发现最优权重衰减随批量大小线性增长，且受数据量/参数比调控。
- 批量大小最优值和临界值均随数据集规模呈幂律增长，与模型大小无关。
- 为多目标训练资源配置提供理论依据，适用于大规模模型部署场景。
高效的大规模语言模型预训练需要精心调优超参数，包括学习率η和权重衰减λ。本文研究超参数的缩放规律：如何随模型规模N、数据集规模D和批量大小B调整这些参数。已有研究指出，AdamW时间尺度τ = B/(ηλD) 应在不同训练设置中保持恒定，我们验证了在固定N和D时，最优λ与B成线性关系。然而，当N和D变化时，我们发现最优τ遵循数据量/参数比D/N的精确幂律。该规律可用于提前准确预测λopt。同时，我们研究了最优批量大小Bopt（使损失最低的批量）和临界批量大小Bcrit（并行训练失效的阈值）的缩放规律。与以往工作不同，我们发现Bopt和Bcrit均随数据集规模D呈幂律增长，与模型大小N无关。最后，分析这些发现如何指导在训练时间和计算资源双重约束下选择帕累托最优的N和D。所有实验均在Cerebras CS-3系统上完成。
原文摘要 · Abstract (English)
Efficient LLM pre-training requires well-tuned hyperparameters (HPs), including learning rate $η$ and weight decay $λ$. We study scaling laws for HPs: formulas for how to scale HPs as we scale model size N, dataset size D, and batch size B. Recent work suggests the AdamW timescale, $τ= B/(ηλD)$, should remain constant across training settings, and we verify the implication that optimal $λ$ scales linearly with B, for a fixed N and D. However, as N and D scale, we show optimal $τ$ obeys a precise power law in the tokens-per-parameter ratio, D/N. This law thus provides a method to accurately predict $λ$opt in advance of large-scale training. We also study scaling laws for optimal batch size Bopt (the B enabling lowest loss at a given N,D) and critical batch size Bcrit (the B beyond which further data parallelism becomes ineffective). In contrast to prior work, we find both Bopt and Bcrit scale as power laws in D, independent of model size, N. Finally, we analyze how these findings inform the real-world selection of Pareto-optimal N and D under dual training time and compute objectives. All experiments were run on Cerebras CS-3 systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。