提出优化器改进方法，可大幅减少甚至消除GPT训练中的学习率预热需求。
Analyzing & Reducing the Need for Learning Rate Warmup in GPT Training
- 通过归一化初始梯度更新方向，抑制过大的参数变动。
- 发现早期梯度更新角度过大是导致需要预热的关键原因。
- 适用于大批次训练的GPT模型，尤其适合追求高效训练的研究者。
学习率预热是大规模批量训练神经网络的常见技巧，但其原理尚不明确。本文聚焦小规模GPT训练中使用AdamW和Lion优化器的情况，分析为何早期梯度更新 $\mathbf{u}_t$ 过大。通过考察 $\ell_2$-范数、方向变化及对网络表示的影响等指标，发现预热的作用在于控制 $Δ\mathbf{w}_t$ 的整体大小，特别是对抗大角度更新。研究还揭示训练初期存在有限的临界批量大小。最终提出一种优化器改进方法，基于上述指标显式归一化 $\mathbf{u}_t$，显著降低甚至消除对学习率预热的需求。
原文摘要 · Abstract (English)
Learning Rate Warmup is a popular heuristic for training neural networks, especially at larger batch sizes, despite limited understanding of its benefits. Warmup decreases the update size $Δ\mathbf{w}_t = η_t \mathbf{u}_t$ early in training by using lower values for the learning rate $η_t$. In this work we argue that warmup benefits training by keeping the overall size of $Δ\mathbf{w}_t$ limited, counteracting large initial values of $\mathbf{u}_t$. Focusing on small-scale GPT training with AdamW/Lion, we explore the following question: Why and by which criteria are early updates $\mathbf{u}_t$ too large? We analyze different metrics for the update size including the $\ell_2$-norm, resulting directional change, and impact on the representations of the network, providing a new perspective on warmup. In particular, we find that warmup helps counteract large angular updates as well as a limited critical batch size early in training. Finally, we show that the need for warmup can be significantly reduced or eliminated by modifying the optimizer to explicitly normalize $\mathbf{u}_t$ based on the aforementioned metrics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。