用统计方法稳定梯度大小，提升大模型训练稳定性
GradientStabilizer:Fix the Norm, Not the Gradient
- 不裁剪梯度方向，改用动态统计估计更新幅度
- 极端梯度峰值时仍能保持参数更新可控，防止发散
- 适配大模型、强化学习等场景，降低对超参敏感度
现代深度学习训练中的不稳定现象常由罕见但极端的梯度范数峰值引发，导致参数更新过大、优化器状态污染，进而出现恢复缓慢或发散。现有梯度裁剪虽可缓解问题，但需调参且无差别截断大更新。本文提出 GradientStabilizer，一种轻量级、可直接替换的梯度变换方法：保留瞬时梯度方向，将更新幅度替换为基于运行梯度范数统计的稳定估计值。理论证明该稳定幅度在峰值步上一致有界，与峰值大小无关，并可控制自适应优化器的状态演化。在大模型预训练（FP16）、量化感知预训练（FP4）、ImageNet分类、强化学习及时间序列预测等多个任务中，该方法均显著提升训练稳定性，扩大稳定学习率范围，并减少发散，甚至大幅降低 Adam 对权重衰减强度的敏感性。代码即将发布。
原文摘要 · Abstract (English)
Training instability in modern deep learning systems is frequently triggered by rare but extreme gradient-norm spikes, which can induce oversized parameter updates, corrupt optimizer state, and lead to slow recovery or divergence. Widely used safeguards such as gradient clipping mitigate these failures but require threshold tuning and indiscriminately truncate large updates. We propose GradientStabilizer, a lightweight, drop-in gradient transform that preserves the instantaneous gradient direction while replacing the update magnitude with a statistically stabilized estimate derived from running gradient-norm statistics. We prove that the resulting stabilized magnitude is uniformly bounded on spike steps, independent of the spike size, and show how this boundedness controls optimizer state evolution in adaptive methods. Across LLM pre-training (FP16), quantization-aware pre-training (FP4), ImageNet classification, reinforcement learning, and time-series forecasting, GradientStabilizer consistently improves training stability, widens stable learning-rate regions, and reduces divergence relative to clipping-based baselines, even substantially reducing Adam's sensitivity to weight-decay strength. Code will be released soon.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。