发现Adam在特定退化多项式上可自动收敛，无需额外调参。
Towards Understanding Adam Convergence on Highly Degenerate Polynomials
- 提出退化多项式下Adam的局部渐近稳定性理论条件。
- 证明Adam在退化函数上实现线性收敛，快于GD和Momentum的亚线性收敛。
- 揭示Adam超参数三类行为模式：稳定收敛、震荡峰值、SignGD式振荡。
Adam是深度学习中广泛使用的优化算法，但其表现出内在优势的具体目标函数类别仍不明确。不同于以往研究需依赖外部调度器且β₂接近1才能收敛，本文探究Adam的“自然”自收敛特性。我们识别出一类高度退化的多项式函数，在该类上Adam可无需额外调度器而自动收敛。具体而言，推导了退化多项式上的局部渐近稳定性理论条件，并验证了理论边界与实验结果的高度一致性。证明了Adam在这些退化函数上达到局部线性收敛，显著优于梯度下降（Gradient Descent）和动量法（Momentum）的亚线性收敛速度。这种加速源于二阶矩 $v_t$ 与平方梯度 $g_t^2$ 之间的解耦机制，使有效学习率呈指数级放大。最后，刻画了Adam的超参数相图，识别出三种不同行为区域：稳定收敛、震荡尖峰、以及类似SignGD的振荡行为。
原文摘要 · Abstract (English)
Adam is a widely used optimization algorithm in deep learning, yet the specific class of objective functions where it exhibits inherent advantages remains underexplored. Unlike prior studies requiring external schedulers and $β_2$ near 1 for convergence, this work investigates the ``natural'' auto-convergence properties of Adam. We identify a class of highly degenerate polynomials where Adam converges automatically without additional schedulers. Specifically, we derive theoretical conditions for local asymptotic stability on degenerate polynomials and demonstrate strong alignment between theoretical bounds and experimental results. We prove that Adam achieves local linear convergence on these degenerate functions, significantly outperforming the sub-linear convergence of Gradient Descent and Momentum. This acceleration stems from a decoupling mechanism between the second moment $v_t$ and squared gradient $g_t^2$, which exponentially amplifies the effective learning rate. Finally, we characterize Adam's hyperparameter phase diagram, identifying three distinct behavioral regimes: stable convergence, spikes, and SignGD-like oscillation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。