提出新方法分析AdaGrad在非凸优化中的收敛性与稳定性。
Stability and convergence analysis of AdaGrad for non-convex optimization via novel stopping time-based techniques
- 用概率论中的新停止时间技术分析算法稳定性
- 证明了几乎必然和均方收敛，且收敛率近最优
- 适合研究优化算法理论的学者参考
自适应梯度优化器（AdaGrad）通过动态调整学习率，在深度学习中表现出色，优于随机梯度下降。然而，其在非凸优化场景下的渐近收敛性和非渐近收敛速率的理论分析仍不充分。本文引入一种来自概率论的新停止时间技术，证明了在温和条件下AdaGrad具有稳定性，并建立了其几乎必然收敛和均方收敛性。此外，我们还推导出平均平方梯度期望下的近最优非渐近收敛率，该结果强于现有高概率结论。本文所发展的技术对其他自适应随机算法的研究具有潜在独立价值。
原文摘要 · Abstract (English)
Adaptive gradient optimizers (AdaGrad), which dynamically adjust the learning rate based on iterative gradients, have emerged as powerful tools in deep learning. These adaptive methods have significantly succeeded in various deep learning tasks, outperforming stochastic gradient descent. However, despite AdaGrad's status as a cornerstone of adaptive optimization, its theoretical analysis has not adequately addressed key aspects such as asymptotic convergence and non-asymptotic convergence rates in non-convex optimization scenarios. This study aims to provide a comprehensive analysis of AdaGrad and bridge the existing gaps in the literature. We introduce a new stopping time technique from probability theory, which allows us to establish the stability of AdaGrad under mild conditions. We further derive the asymptotically almost sure and mean-square convergence for AdaGrad. In addition, we demonstrate the near-optimal non-asymptotic convergence rate measured by the average-squared gradients in expectation, which is stronger than the existing high-probability results. The techniques developed in this work are potentially of independent interest for future research on other adaptive stochastic algorithms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。