通过激活缩放提升大模型预训练收敛速度，不损失梯度信息。
GPAS: Accelerating Convergence of LLM Pretraining via Gradient-Preserving Activation Scaling
- 引入梯度保持的激活缩放机制，缓解深层网络激活方差爆炸问题。
- 在71M至1B参数模型上均实现稳定性能提升，最大加速比达1.5倍。
- 适用于Pre-LN、Sandwich-LN等主流架构，适合大规模语言模型开发者。
当前主流大语言模型如LLaMA、Qwen和DeepSeek系列普遍采用Pre-LayerNorm（Pre-LN）Transformer结构。尽管该结构在预训练中表现稳定且可扩展至大模型规模，但其存在激活方差随层数指数增长的问题，导致残差连接中捷径项主导子层输出，限制了深层网络的学习能力。为此，我们提出梯度保持的激活缩放（GPAS）技术，可在不改变梯度的前提下缩小中间激活值。该方法保留了激活中的信息，避免了梯度缩放带来的消失问题。在71M到1B参数量级的多种模型上进行的广泛实验表明，GPAS实现了持续的性能提升。此外，该方法在Sandwich-LN和DeepNorm等其他架构中也展现出改进潜力，证明其在多种训练场景下的通用性和有效性。代码已开源：https://github.com/dandingsky/GPAS。
原文摘要 · Abstract (English)
Modern Large Language Models, such as the LLaMA, Qwen and DeepSeek series, predominantly adopt the Pre-LayerNorm (Pre-LN) Transformer architecture. While being stable during pretraining and scalable to large model sizes, Pre-LN suffers from an exponential growth in activation variance across layers, causing the shortcut to dominate over sub-layer outputs in the residual connection and limiting the learning capacity of deeper layers. To mitigate this issue, we propose Gradient-Preserving Activation Scaling (GPAS), a simple technique that can be used in combination with existing approaches. GPAS works by scaling down the intermediate activations while keeping their gradients unchanged. This leaves information in the activations intact, and avoids the gradient vanishing problem associated with gradient downscaling. Extensive experiments across various model sizes from 71M to 1B show that GPAS achieves consistent performance gains. Beyond enhancing Pre-LN Transformers, GPAS also shows promise in improving alternative architectures such as Sandwich-LN and DeepNorm, demonstrating its versatility and potential for improving training dynamics in a wide range of settings. Our code is available at https://github.com/dandingsky/GPAS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。