提出可连续调节适应性的新优化器Anon,突破SGD与Adam局限。
Anon: Extrapolating Adaptivity Beyond SGD and Adam

- 设计可连续调节适应性的优化器,自由切换类SGD与类Adam行为
- 在图像分类、扩散模型和语言建模任务上全面超越现有优化器
- 提出增量延迟更新机制,提升对梯度噪声的鲁棒性
自适应优化器如Adam在训练大规模模型(如大语言模型和扩散模型)中表现优异,但其泛化性能常弱于非自适应方法(如SGD)。我们识别出性能差距的关键原因:预条件器中的自适应性限制了优化器对多样化优化景观的适应能力。为此,我们提出Anon(Adaptivity Non-restricted Optimizer with Novel convergence technique),一种可在R中连续调节适应性的新型优化器,可插值于类SGD与类Adam行为之间,甚至超越两者。为确保整个适应性范围内的收敛性,引入增量延迟更新(IDU)机制,其灵活性优于AMSGrad的硬最大跟踪策略,并增强对梯度噪声的鲁棒性。理论证明了在凸与非凸设置下的收敛性。实验显示,Anon在图像分类、扩散模型和语言建模等代表性任务上持续优于当前最优优化器。结果表明,适应性可作为可调的设计原则,Anon首次提供统一且可靠的框架,弥合经典与现代优化器的差距并超越其优势。
原文摘要 · Abstract (English)
Adaptive optimizers such as Adam have achieved great success in training large-scale models like large language models and diffusion models. However, they often generalize worse than non-adaptive methods, such as SGD on classical architectures like CNNs. We identify a key cause of this performance gap: adaptivity in pre-conditioners, which limits the optimizer's ability to adapt to diverse optimization landscapes. To address this, we propose Anon (Adaptivity Non-restricted Optimizer with Novel convergence technique), a novel optimizer with continuously tunable adaptivity in R, allowing it to interpolate between SGD-like and Adam-like behaviors and even extrapolate beyond both. To ensure convergence across the entire adaptivity spectrum, we introduce incremental delay update (IDU), a novel mechanism that is more flexible than AMSGrad's hard max-tracking strategy and enhances robustness to gradient noise. We theoretically establish convergence guarantees under both convex and non-convex settings. Empirically, Anon consistently outperforms state-of-the-art optimizers on representative image classification, diffusion, and language modeling tasks. These results demonstrate that adaptivity can serve as a valuable tunable design principle, and Anon provides the first unified and reliable framework capable of bridging the gap between classical and modern optimizers and surpassing their advantageous properties.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。