用高速公路连接重拾后层归一化，让千层大模型稳定训练
Post-LayerNorm Is Back: Stable, ExpressivE, and Deep
- 改用高速路连接替代残差路径，解决深层网络梯度消失问题
- 在1000层以上深度训练稳定，困惑度持续下降
- 无需特殊初始化，适合构建超深语言模型
大型语言模型（LLM）的扩展正遇瓶颈：拓宽模型收益递减，延长上下文也难提升表达能力。相比之下，深度扩展理论上更具优势，但现有Transformer架构在极端深度下难以稳定训练。本文重新审视曾被现代LLM弃用的后层归一化（Post-LN）结构。我们发现其失败主因在于ResNet式残差路径导致深层梯度消失。为此提出Keel，一种采用高速路（Highway-style）连接替代残差路径的Post-LN Transformer。该设计保持残差分支的梯度流动，有效防止信号从顶层向底层衰减。与以往方法不同，Keel无需特殊初始化或复杂优化技巧，即可在超过1000层的深度上稳定训练，并持续优于Pre-LN的困惑度与深度扩展特性。结果表明，结合高速路连接的Post-LN为构建可深度扩展的LLM提供了简单而有效的基础，为未来无限深度架构开辟可能。
原文摘要 · Abstract (English)
Large language model (LLM) scaling is hitting a wall. Widening models yields diminishing returns, and extending context length does not improve fundamental expressivity. In contrast, depth scaling offers theoretically superior expressivity, yet current Transformer architectures struggle to train reliably at extreme depths. We revisit the Post-LayerNorm (Post-LN) formulation, whose instability at scale caused its replacement by Pre-LN in modern LLMs. We show that the central failure mode of Post-LN arises from the ResNet-style residual pathway, which introduces gradient vanishing in deep networks. We present Keel, a Post-LN Transformer that replaces this residual path with a Highway-style connection. This modification preserves the gradient flow through the residual branch, preventing signal vanishing from the top layers to the bottom. Unlike prior methods, Keel enables stable training at extreme depths without requiring specialized initialization or complex optimization tricks. Keel trains robustly at depths exceeding 1000 layers and consistently improves perplexity and depth-scaling characteristics over Pre-LN. These findings indicate that Post-LN, when paired with a Highway-style connection, provides a simple and effective foundation for building deeply scalable LLMs, opening the possibility for future infinite-depth architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。