Grams通过分离梯度方向与动量幅度，实现更优的训练速度和泛化性能。
Grams: Gradient Descent with Adaptive Momentum Scaling
- 将梯度方向与动量幅度解耦，动量仅用于自适应调节更新大小。
- 在多个任务上收敛更快，且比Adam、Lion等优化器泛化能力更强。
- 适用于大模型高效训练与微调，代码已开源。
我们提出了一种新的优化算法Grams（Gradient Descent with Adaptive Momentum Scaling），其核心思想是将参数更新的方向（由当前梯度决定）与动量（仅用于自适应调节更新幅度）解耦。与传统优化器直接融合动量不同，Grams独立处理方向与幅度，从而实现更优的损失下降。理论上证明Grams比现有先进优化器下降速度更快，并建立了全局收敛性保证。大量实验验证了其有效性：在多个基准任务上，Grams展现出更快的收敛速度和更好的泛化性能，显著优于广泛使用的Adam、Lion及其谨慎变体。结果表明Grams在高效训练与微调大规模语言模型方面具有巨大潜力。代码已公开于https://github.com/Gunale0926/Grams。
原文摘要 · Abstract (English)
We introduce $\mathbf{G}$radient Descent with $\mathbf{A}$daptive $\mathbf{M}$omentum $\mathbf{S}$caling ($\mathbf{Grams}$), a novel optimization algorithm that decouples the direction and magnitude of parameter updates in deep learning. Unlike traditional optimizers that directly integrate momentum into updates, Grams separates the update direction, derived from current gradients, from momentum, which is used solely for adaptive magnitude scaling. This approach enables Grams to achieve improved loss descent compared to state-of-the-art cautious and momentum-based optimizers. We theoretically demonstrate that Grams descents faster than other state-of-the-art optimizers and establish a global convergence guarantee for Grams. We also validate its effectiveness through extensive empirical evaluations. The results demonstrate Grams' superior performance, including faster convergence and better generalization, compared to widely-used optimizers such as Adam, Lion, and their cautious variants. Our results highlight Grams' potential as a transformative approach for efficiently training and fine-tuning large language models. Code is available at https://github.com/Gunale0926/Grams.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。