发现最优训练参数由输出层范数决定，可统一指导大模型训练。
Optimal Scaling Needs Optimal Norm
- 用输出层算子范数统一解释跨模型和数据规模的最优参数
- 在1.3B参数、138B词元数据上，最优学习率与批量大小的范数恒定
- 适合研究大模型训练规律或优化器设计的科研人员
尽管在模型与数据规模下的最优超参数迁移方面取得进展，但尚未建立统一解释原则。针对Adam和Scion优化器，我们发现跨模型与数据规模的联合最优缩放依赖于单一不变量：输出层的算子范数。在最大1.3B参数、138B词元数据的模型上，最优学习率与批量大小组合(η∗, B∗)始终具有相同的算子范数值——这一现象称为范数迁移。该常数范数条件是必要但非充分的：对每个数据集规模，多个(η, B)可达最优范数，但仅唯一一组(η∗, B∗)能实现最低损失。作为充分条件，我们首次测量了Scion优化器下(η∗, B∗)随数据规模的缩放规律，结果与Adam一致。调整分层组学习率亦可提升性能，其中输出层最敏感，隐藏层采用较低学习率更优。本文提供基于范数的最优缩放实践建议，并开源分布式Scion（Disco）实现及超过两千次运行日志，支持大规模大模型训练动态研究。
原文摘要 · Abstract (English)
Despite recent progress in optimal hyperparameter transfer under model and dataset scaling, no unifying explanatory principle has been established. For Adam and Scion optimizers, we discover that joint optimal scaling across model and dataset sizes is conditioned on a single invariant: the operator norm of the output layer. Across models with up to 1.3B parameters trained on up to 138B tokens, the optimal learning rate/batch size pair $(η^{\ast}, B^{\ast})$ consistently has the same operator norm value - a phenomenon we term norm transfer. This constant norm condition is necessary but not sufficient: while for each dataset size, multiple $(η, B)$ reach the optimal norm, only a unique $(η^{\ast}, B^{\ast})$ achieves the best loss. As a sufficient condition, we provide the first measurement of $(η^{\ast}, B^{\ast})$ scaling with dataset size for Scion, and find that the scaling rules are consistent with those of Adam. Tuning per-layer-group learning rates also improves model performance, with the output layer being the most sensitive and hidden layers benefiting from lower learning rates. We provide practical insights on norm-guided optimal scaling and release our Distributed Scion (Disco) implementation with logs from over two thousand runs to support research on LLM training dynamics at scale.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。