提出SpanNorm,让Transformer既稳定又高效。
SpanNorm: Reconciling Training Stability and Performance in Deep Transformers
- 用跨块残差连接+后归一化计算,融合两种归一化优势。
- 理论证明信号方差可控,训练不崩溃,性能不退化。
- 在密集和MoE模型中均超越传统方法,适合深度模型训练。
大型语言模型的成功依赖于深层Transformer架构的稳定训练。归一化层的位置选择带来根本性权衡:预归一化(PreNorm)保障训练稳定但深层模型易性能下降,后归一化(PostNorm)表现强却严重不稳定。本文提出SpanNorm,通过建立贯穿整个Transformer块的清晰残差连接,稳定信号传播;同时采用后归一化风格计算,提升模型性能。理论分析表明,结合合理缩放策略,SpanNorm可保持网络内信号方差有界,避免后归一化梯度问题,并缓解预归一化表示坍塌。实验显示,SpanNorm在密集模型与混合专家(MoE)场景中持续优于标准归一化方案,为更强大且稳定的Transformer架构铺平道路。
原文摘要 · Abstract (English)
The success of Large Language Models (LLMs) hinges on the stable training of deep Transformer architectures. A critical design choice is the placement of normalization layers, leading to a fundamental trade-off: the ``PreNorm'' architecture ensures training stability at the cost of potential performance degradation in deep models, while the ``PostNorm'' architecture offers strong performance but suffers from severe training instability. In this work, we propose SpanNorm, a novel technique designed to resolve this dilemma by integrating the strengths of both paradigms. Structurally, SpanNorm establishes a clean residual connection that spans the entire transformer block to stabilize signal propagation, while employing a PostNorm-style computation that normalizes the aggregated output to enhance model performance. We provide a theoretical analysis demonstrating that SpanNorm, combined with a principled scaling strategy, maintains bounded signal variance throughout the network, preventing the gradient issues that plague PostNorm models, and also alleviating the representation collapse of PreNorm. Empirically, SpanNorm consistently outperforms standard normalization schemes in both dense and Mixture-of-Experts (MoE) scenarios, paving the way for more powerful and stable Transformer architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。