揭示Adam优化器中β₁与β₂的最优设置关系。
How to Set $β_1, β_2$ in Adam: An Online Learning Perspective
- 从在线学习视角重新分析Adam，突破传统β₁=√β₂限制。
- 证明β₁=√β₂对静止对手最优，但对动态对手次优。
- 给出更普适的理论边界，适用于实际训练中的任意β₁、β₂组合。
尽管Adam是训练大规模机器学习模型最有效的优化器之一，但对其动量因子β₁和β₂的最优设置仍缺乏理论理解。已有研究将Adam视为在线学习中重要的Follow-the-Regularized-Leader（FTRL）算法实例，但其分析要求β₁ = √β₂，无法覆盖β₁ ≠ √β₂的实用场景。本文推导出更通用的新分析，适用于β₁ ≥ √β₂和β₁ ≤ √β₂两种情况，结果严格推广了现有界限。此外，我们证明这些界限在最坏情况下是紧的。研究还表明，β₁ = √β₂对静止对手（oblivious adversary）最优，但对非静止对手（non-oblivious adversary）则次优。
原文摘要 · Abstract (English)
While Adam is one of the most effective optimizer for training large-scale machine learning models, a theoretical understanding of how to optimally set its momentum factors, $β_1$ and $β_2$, remains largely incomplete. Prior works have shown that Adam can be seen as an instance of Follow-the-Regularized-Leader (FTRL), one of the most important class of algorithms in online learning. The prior analyses in these works required setting $β_1 = \sqrt{β_2}$, which does not cover the more practical cases with $β_1 \neq \sqrt{β_2}$. We derive novel, more general analyses that hold for both $β_1 \geq \sqrt{β_2}$ and $β_1 \leq \sqrt{β_2}$. In both cases, our results strictly generalize the existing bounds. Furthermore, we show that our bounds are tight in the worst case. We also prove that setting $β_1 = \sqrt{β_2}$ is optimal for an oblivious adversary, but sub-optimal for an non-oblivious adversary.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。