提出连续时间模型分析无放回随机梯度下降，证明其收敛性并给出更优上界。
Towards Continuous-Time Approximations for Stochastic Gradient Descent without Replacement
- 用带周期布朗运动的随机微分方程建模无放回梯度下降
- 证明强凸目标下学习率1/(1+t)^β时几乎必然收敛
- 给出收敛速度上界，优于已有结果，适合优化理论研究者
实践中主流的梯度优化算法采用多轮训练（即无放回随机梯度下降，SGDo），但其数学理论远未成熟于有放回或单遍算法。本文提出一种基于“周期布朗运动”的随机连续时间逼近模型，通过年轻型微分方程描述带噪声的SGDo。我们证明了在强凸目标函数下，学习率形式为 $u_t = \frac{1}{(1+t)^β}$（$β\in (0,1)$）时，该连续时间近似几乎必然收敛。进一步计算出渐近收敛速度的上界，其性能不低于甚至优于现有针对SGDo的结果。
原文摘要 · Abstract (English)
Gradient optimization algorithms using epochs, that is those based on stochastic gradient descent without replacement (SGDo), are predominantly used to train machine learning models in practice. However, the mathematical theory of SGDo and related algorithms remain underexplored compared to their "with replacement" and "one-pass" counterparts. In this article, we propose a stochastic, continuous-time approximation to SGDo with additive noise based on a Young differential equation driven by a stochastic process we call an "epoched Brownian motion". We show its usefulness by proving the almost sure convergence of the continuous-time approximation for strongly convex objectives and learning rate schedules of the form $u_t = \frac{1}{(1+t)^β}, β\in (0,1)$. Moreover, we compute an upper bound on the asymptotic rate of almost sure convergence, which is as good or better than previous results for SGDo.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。