通过解耦权重尺度与分布，稳定大模型训练过程。
Scale-Distribution Decoupling: Enabling Stable and Effective Training of Large Language Models
- 将权重的尺度和分布分开处理，提升训练稳定性。
- 有效防止梯度爆炸与消失，尤其在深层网络中表现优异。
- 轻量且兼容现有框架，适合实际大模型训练使用。
大语言模型预训练中的训练稳定性是长期挑战，尤其对于后归一化（Post-Norm）Transformer架构，易出现梯度爆炸与消失问题。本文提出尺度-分布解耦（Scale-Distribution Decoupling, SDD），通过显式解耦全连接层权重矩阵的尺度与分布，利用归一化机制调控激活值，并引入可学习缩放向量以维持良好条件的梯度，从而有效防止梯度爆炸与消失。该分离策略提升了优化效率，尤其在深层网络中确保了稳定的梯度传播。实验表明，该方法在多种大模型架构及不同归一化配置下均能稳定训练，性能优于现有技术。此外，SDD实现轻量，兼容现有训练框架，具备实际应用价值。代码已开源：https://github.com/kaihemo/SDD。
原文摘要 · Abstract (English)
Training stability is a persistent challenge in the pre-training of large language models (LLMs), particularly for architectures such as Post-Norm Transformers, which are prone to gradient explosion and dissipation. In this paper, we propose Scale-Distribution Decoupling (SDD), a novel approach that stabilizes training by explicitly decoupling the scale and distribution of the weight matrix in fully-connected layers. SDD applies a normalization mechanism to regulate activations and a learnable scaling vector to maintain well-conditioned gradients, effectively preventing $\textbf{gradient explosion and dissipation}$. This separation improves optimization efficiency, particularly in deep networks, by ensuring stable gradient propagation. Experimental results demonstrate that our method stabilizes training across various LLM architectures and outperforms existing techniques in different normalization configurations. Furthermore, the proposed method is lightweight and compatible with existing frameworks, making it a practical solution for stabilizing LLM training. Code is available at https://github.com/kaihemo/SDD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。