通过梯度符号幂变换加速大模型预训练，无需修改优化器。
GradPower: Powering Gradients for Faster Language Model Pre-Training
- 对梯度向量应用符号幂变换，提升优化效率。
- 在多种模型与数据集上实现更低的最终损失，尤其适合MoE模型。
- 代码改动极小，兼容主流优化器，适合追求训练速度的研究者。
我们提出GradPower，一种轻量级梯度变换技术，用于加速语言模型预训练。给定梯度向量 $g=(g_i)_i$，GradPower首先进行逐元素符号幂变换：$φ_p(g)=({\rm sign}(g_i)|g_i|^p)_{i}$（固定 $p>0$），然后将变换后的梯度输入基础优化器。值得注意的是，GradPower只需一行代码变更，且不改变基础优化器的内部逻辑或超参数。将其应用于Adam（称为AdamPower）时，在多种架构（LLaMA、Qwen2MoE）、参数规模（66M至2B）、数据集（C4、OpenWebText）和学习率调度（余弦、预热-稳定-衰减）下均实现更低的终端损失。最显著的提升出现在使用预热-稳定-衰减调度训练现代混合专家模型时。GradPower还能无缝集成其他前沿优化器（如Muon），进一步提升性能。最后，我们提供了理论分析，揭示了其底层机制并强调梯度噪声的影响。
原文摘要 · Abstract (English)
We propose GradPower, a lightweight gradient-transformation technique for accelerating language model pre-training. Given a gradient vector $g=(g_i)_i$, GradPower first applies the elementwise sign-power transformation: $φ_p(g)=({\rm sign}(g_i)|g_i|^p)_{i}$ for a fixed $p>0$, and then feeds the transformed gradient into a base optimizer. Notably, GradPower requires only a single-line code change and no modifications to the base optimizer's internal logic, including the hyperparameters. When applied to Adam (termed AdamPower), GradPower consistently achieves lower terminal loss across diverse architectures (LLaMA, Qwen2MoE), parameter scales (66M to 2B), datasets (C4, OpenWebText), and learning-rate schedules (cosine, warmup-stable-decay). The most pronounced gains are observed when training modern mixture-of-experts models with warmup-stable-decay schedules. GradPower also integrates seamlessly with other state-of-the-art optimizers, such as Muon, yielding further improvements. Finally, we provide theoretical analyses that reveal the underlying mechanism of GradPower and highlight the influence of gradient noise.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。