发现大模型训练末期梯度突增原因并提出简单修复方法

Why Gradients Rapidly Increase Near the End of Training

• 揭示权重衰减、归一化层与学习率调度的意外交互导致梯度飙升
• 修正后训练全程损失更低，且避免末期梯度异常增长
• 对长周期LLM训练有实用价值，适合关注训练稳定性研究者

在长时间的大语言模型（LLM）训练过程中，梯度范数在训练末期会迅速增加。本文指出，这一现象源于权重衰减、归一化层与学习率调度之间的非预期相互作用。我们提出一种简单修正方法，可有效消除该问题，同时使整个训练过程中的损失值更低。该方法无需修改模型结构，仅调整优化器行为即可实现。实验表明，修正后的训练过程更稳定，收敛性能更优。

原文摘要 · Abstract (English)

During long-duration Large Language Model (LLM) training runs the gradient norm increases rapidly near the end of training. In this short note, we show that this increase is due to an unintended interaction between weight decay, normalization layers, and the learning rate schedule. We propose a simple correction that fixes this behavior while also resulting in lower loss values throughout training.

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。