提出新型归一化位置Peri-LN，提升大模型训练稳定性
Peri-LN: Revisiting Normalization Layer in the Transformer Architecture
- 将归一化层置于子层外围，改善激活值方差与梯度传播
- 在32亿参数模型上实现更均衡的方差增长和稳定收敛
- 适合追求训练稳定的大型Transformer架构设计
选择能稳定训练并加速收敛的层归一化（LN）策略，对当今的大规模语言模型仍具挑战。本文构建了理解不同LN策略如何影响大规模Transformer训练动态的分析基础。尽管此前预归一化（Pre-LN）和后归一化（Post-LN）长期主导实践，但近期多个开源模型悄然采用第三种策略：将归一化层置于子层外围，我们称之为Peri-LN。尽管其表现优异，但其机制与优势仍几乎未被研究。通过深入分析，我们揭示了不同放置方式对激活方差与梯度传播的影响。在高达32亿参数的Transformer上进行广泛实验，结果表明Peri-LN始终实现更平衡的方差增长、更平稳的梯度流动与更高的收敛稳定性。研究建议在大规模Transformer中应更广泛考虑Peri-LN，为归一化层最优布局提供新洞见。
原文摘要 · Abstract (English)
Selecting a layer normalization (LN) strategy that stabilizes training and speeds convergence in Transformers remains difficult, even for today's large language models (LLM). We present a comprehensive analytical foundation for understanding how different LN strategies influence training dynamics in large-scale Transformers. Until recently, Pre-LN and Post-LN have long dominated practices despite their limitations in large-scale training. However, several open-source models have recently begun silently adopting a third strategy without much explanation. This strategy places normalization layer peripherally around sublayers, a design we term Peri-LN. While Peri-LN has demonstrated promising performance, its precise mechanisms and benefits remain almost unexplored. Our in-depth analysis delineates the distinct behaviors of LN strategies, showing how each placement shapes activation variance and gradient propagation. To validate our theoretical insight, we conduct extensive experiments on Transformers up to $3.2$B parameters, showing that Peri-LN consistently achieves more balanced variance growth, steadier gradient flow, and convergence stability. Our results suggest that Peri-LN warrants broader consideration for large-scale Transformer architectures, providing renewed insights into the optimal placement of LN.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。