提出可调步长的λ-SAGA算法，提升收敛性分析精度
On the SAGA algorithm with decreasing step
- 设计λ-插值算法，融合SGD与SAGA特性
- 在非强凸和非Lipschitz梯度下实现几乎必然收敛
- 给出L^p非渐近收敛率，适用于复杂优化场景
随机优化广泛应用于机器学习等领域。本文提出一种新的λ-SAGA算法，该算法在随机梯度下降（λ=0）与SAGA算法（λ=1）之间进行插值。首先，在递减步长下研究该算法的几乎必然收敛性，避免了目标函数强凸性和Lipschitz梯度的严格假设。其次，建立了λ-SAGA算法的中心极限定理。最后，给出了该算法的非渐近L^p收敛速率。结果拓展了SAGA算法的理论边界，为非标准条件下的优化提供了新工具。
原文摘要 · Abstract (English)
Stochastic optimization naturally appear in many application areas, including machine learning. Our goal is to go further in the analysis of the Stochastic Average Gradient Accelerated (SAGA) algorithm. To achieve this, we introduce a new $λ$-SAGA algorithm which interpolates between the Stochastic Gradient Descent ($λ=0$) and the SAGA algorithm ($λ=1$). Firstly, we investigate the almost sure convergence of this new algorithm with decreasing step which allows us to avoid the restrictive strong convexity and Lipschitz gradient hypotheses associated to the objective function. Secondly, we establish a central limit theorem for the $λ$-SAGA algorithm. Finally, we provide the non-asymptotic $\mathbb{L}^p$ rates of convergence.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。