通过简单归一化提升GPT训练稳定性与性能
SimpleGPT: Improving GPT via A Simple Normalization Strategy
- 引入SimpleNorm归一化策略，稳定中间激活值尺度
- 实验显示可支持3到10倍更大的学习率，7B模型训练损失降低0.08
- 适合追求高效训练和高精度的大型语言模型研究者
本文从二阶几何角度重新审视Transformer优化，建立架构设计、激活尺度、海森矩阵与最大容忍学习率之间的直接联系。提出一种名为SimpleNorm的简单归一化策略，通过构造方式稳定中间激活尺度。分析损失关于网络激活的海森矩阵后，理论证明SimpleNorm显著降低海森矩阵的谱范数，从而允许更大且稳定的训练学习率。在1B、1.4B、7B和8B参数规模的大型GPT模型上进行广泛实验验证。结果表明，基于SimpleNorm的SimpleGPT模型可容忍比常规方法大3至10倍的学习率，优化过程高度稳定，并显著优于现有基线。例如，在60K步训练7B模型时，其训练损失为2.208，较采用QKNorm的LLaMA2（2.290）降低0.08。
原文摘要 · Abstract (English)
In this work, we revisit Transformer optimization through the lens of second-order geometry and establish a direct connection between architectural design, activation scale, the Hessian matrix, and the maximum tolerable learning rate. We introduce a simple normalization strategy, termed SimpleNorm, which stabilizes intermediate activation scales by construction. Then, by analyzing the Hessian of the loss with respect to network activations, we theoretically show that SimpleNorm significantly reduces the spectral norm of the Hessian, thereby permitting larger stable learning rates. We validate our theoretical findings through extensive experiments on large GPT models at parameter scales 1B, 1.4B, 7B and 8B. Empirically, SimpleGPT, our SimpleNorm-based network, tolerates learning rates 3$\times$-10$\times$ larger than standard convention, consistently demonstrates strong optimization stability, and achieves substantially better performance than well-established baselines. Specifically, when training 7B-scale models for 60K steps, SimpleGPT achieves a training loss that is 0.08 lower than that of LLaMA2 with QKNorm, reducing the loss from 2.290 to 2.208. Our source code will be released at https://github.com/Ocram7/SimpleGPT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。