ADOPT让Adam在任意β₂下都能收敛，且速度最优。
ADOPT: Modified Adam Can Converge with Any $β_2$ with the Optimal Rate
- 移除梯度二阶矩估计中的当前梯度，调整更新顺序。
- 理论证明可实现1/√T的最优收敛率，无需梯度噪声有界假设。
- 适用广泛任务，性能优于Adam及其变体，适合追求稳定训练的实践者。
Adam是深度学习中最流行的优化算法之一，但理论上只有在特定选择超参数β₂时才能收敛。已有多种改进方法（如AMSGrad）试图解决此问题，但均依赖于梯度噪声有界的不切实际假设。本文提出新型自适应梯度方法ADOPT，可在任意β₂取值下实现最优收敛率O(1/√T)，且无需依赖梯度噪声有界假设。ADOPT通过从二阶矩估计中移除当前梯度，并改变动量更新与归一化顺序来解决非收敛问题。我们进行了大量数值实验，验证了ADOPT在图像分类、生成建模、自然语言处理和深度强化学习等任务中均优于Adam及其变体。代码已开源：https://github.com/iShohei220/adopt。
原文摘要 · Abstract (English)
Adam is one of the most popular optimization algorithms in deep learning. However, it is known that Adam does not converge in theory unless choosing a hyperparameter, i.e., $β_2$, in a problem-dependent manner. There have been many attempts to fix the non-convergence (e.g., AMSGrad), but they require an impractical assumption that the gradient noise is uniformly bounded. In this paper, we propose a new adaptive gradient method named ADOPT, which achieves the optimal convergence rate of $\mathcal{O} ( 1 / \sqrt{T} )$ with any choice of $β_2$ without depending on the bounded noise assumption. ADOPT addresses the non-convergence issue of Adam by removing the current gradient from the second moment estimate and changing the order of the momentum update and the normalization by the second moment estimate. We also conduct intensive numerical experiments, and verify that our ADOPT achieves superior results compared to Adam and its variants across a wide range of tasks, including image classification, generative modeling, natural language processing, and deep reinforcement learning. The implementation is available at https://github.com/iShohei220/adopt.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。