通过渐进式残差暖化提升大模型预训练稳定性与速度
Progressive Residual Warmup for Language Model Pretraining
- 深层网络延迟激活，让浅层先稳定再逐步参与学习
- 预训练收敛更快，下游任务性能提升3%-5%以上
- 适合大规模语言模型训练，尤其对深层架构有效
Transformer 架构是现代大语言模型的核心，其预训练的稳定性和收敛速度至关重要。受堆叠层间逻辑依赖的启发，本文提出渐进式残差暖化（ProRes）方法。该方法通过为每层残差乘以从0逐步升至1的缩放因子，实现‘浅层先学’的理念，深层网络在更晚阶段才开始参与学习，从而等待浅层进入更稳定的状态。我们在多种模型规模、归一化和初始化方案下进行预训练实验，结果表明，ProRes不仅显著提升了预训练稳定性，还带来了独特的优化轨迹，实现更快收敛、更强泛化能力及更好的下游性能。代码已开源：https://github.com/dandingsky/ProRes。
原文摘要 · Abstract (English)
Transformer architectures serve as the backbone for most modern Large Language Models, therefore their pretraining stability and convergence speed are of central concern. Motivated by the logical dependency of sequentially stacked layers, we propose Progressive Residual Warmup (ProRes) for language model pretraining. ProRes implements an "early layer learns first" philosophy by multiplying each layer's residual with a scalar that gradually warms up from 0 to 1, with deeper layers taking longer warmup steps. In this way, deeper layers wait for early layers to settle into a more stable regime before contributing to learning. We demonstrate the effectiveness of ProRes through pretraining experiments across various model scales, as well as normalization and initialization schemes. Comprehensive analysis shows that ProRes not only stabilizes pretraining but also introduces a unique optimization trajectory, leading to faster convergence, stronger generalization and better downstream performance. Our code is available at https://github.com/dandingsky/ProRes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。