仅改一行代码，让优化器训练更快更稳。
Cautious Optimizers: Improving Training with One Line of Code
- 在动量类优化器中加入一行为改进，无需重写核心逻辑。
- 在大模型预训练和图像分类任务中均实现稳定加速，超参调整极少。
- 理论证明保留收敛性，揭示新优化器家族，适合追求效率的研究者。
AdamW 是变压器模型预训练的默认优化器。多年来，社区一直在寻找更快更稳定的优化器，但成效有限。本文提出对任何基于动量的优化器进行一行为修改，称为谨慎优化器（Cautious Optimizer），如 C-AdamW、C-Lion。理论分析表明，该修改保持了 Adam 的哈密顿函数，并在李雅普诺夫分析下不破坏收敛性保证。此外，这一理论洞察揭示了一个全新的优化器家族。我们从中选取最简单的形式进行实验，结果在大语言模型预训练和图像分类任务中均实现一致提速，且超参数调优极少。代码已开源：https://github.com/kyleliang919/C-Optim。
原文摘要 · Abstract (English)
AdamW has been the default optimizer for transformer pretraining. For many years, our community searched for faster and more stable optimizers with only constrained positive outcomes. In this work, we propose a \textbf{one-line modification in Pytorch} to any momentum-based optimizer, which we rename cautious optimizer, e.g. C-AdamW and C-Lion. Our theoretical result shows that this modification preserves Adam's Hamiltonian function and it does not break the convergence guarantee under the Lyapunov analysis. In addition, a whole new family of optimizers is revealed by our theoretical insight. Among them, we pick the simplest one for empirical experiments, showing not only consistent speed-up on LLM pretraining, but also image classification, with minimum extra tuning on hyperparameters. Code is available at https://github.com/kyleliang919/C-Optim.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。