提出HomeAdam，让Adam优化器偶尔回归SGD，提升泛化性能。
HomeAdam: Adam and AdamW Algorithms Sometimes Go Home to Obtain Better Provable Generalization
- 让Adam在训练中定期切换回SGD式更新，改善泛化能力。
- 理论证明泛化误差降为1/N，优于原版Adam的1/√N。
- 实验验证新算法收敛快、泛化好，适合追求精度的深度学习任务。
Adam和AdamW是深度学习中默认的自适应优化器，虽收敛快但泛化性能较差，其理论泛化误差为O(1/√N)，高于SGD的O(1/N)，其中N为训练样本数。本文通过算法稳定性重新分析了Adam与AdamW的泛化性，首次证明无平方根版本（Adam(W)-srf）的泛化误差为O(ρ̂⁻²ᵀ / N)，ρ̂为二阶动量最小值加小正数，通常极小。为此，我们提出一类高效智能的HomeAdam(W)算法，通过在训练中有时返回基于动量的SGD来改进泛化。理论上，HomeAdam(W)的泛化误差为O(1/N)，优于Adam(W)-srf的O(ρ̂⁻²ᵀ / N)，且快于现有O(1/√N)的Adam(W)。同时，其收敛速度达O(1/T¹/⁴)，优于Adam(W)-srf的O(ρ̌⁻¹ / T¹/⁴)，其中ρ̌ ≤ ρ̂也极小。大量数值实验验证了HomeAdam(W)的有效性。
原文摘要 · Abstract (English)
Adam and AdamW are a class of default optimizers for training deep learning models in machine learning. These adaptive algorithms converge faster but generalize worse compared to SGD. In fact, their proved generalization error $O(\frac{1}{\sqrt{N}})$ also is larger than $O(\frac{1}{N})$ of SGD, where $N$ denotes training sample size. Recently, although some variants of Adam have been proposed to improve its generalization, their improved generalizations are still unexplored in theory. To fill this gap, in the paper, we restudy generalization of Adam and AdamW via algorithmic stability, and first prove that Adam and AdamW without square-root (i.e., Adam(W)-srf) have a generalization error $O(\frac{\hatρ^{-2T}}{N})$, where $T$ denotes iteration number and $\hatρ>0$ denotes the smallest element of second-order momentum plus a small positive number. To improve generalization, we propose a class of efficient clever Adam (i.e., HomeAdam(W)) algorithms via sometimes returning momentum-based SGD. Moreover, we prove that our HomeAdam(W) have a smaller generalization error $O(\frac{1}{N})$ than $O(\frac{\hatρ^{-2T}}{N})$ of Adam(W)-srf, since $\hatρ$ is generally very small. In particular, it is also smaller than the existing $O(\frac{1}{\sqrt{N}})$ of Adam(W). Meanwhile, we prove our HomeAdam(W) have a faster convergence rate of $O(\frac{1}{T^{1/4}})$ than $O(\frac{\breveρ^{-1}}{T^{1/4}})$ of the Adam(W)-srf, where $\breveρ\leq\hatρ$ also is very small. Extensive numerical experiments demonstrate efficiency of our HomeAdam(W) algorithms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。