分析随机梯度下降最后迭代点的收敛速度，给出精确率结果。
Convergence Rate for the Last Iterate of Stochastic Gradient Descent Schemes
- 用离散格朗沃尔不等式分析收敛性，避免复杂定理依赖。
- 非凸目标下梯度范数趋于零的速度为 o(t^{p-1})。
- 适用于关注优化算法最后一步性能的研究者。
研究在参数设置下，当目标函数 $F$ 全局凸或梯度满足 $γ$-霍尔德连续的非凸函数时，随机梯度下降（SGD）和随机重球法（SHB）最后迭代点的收敛速率。仅使用离散格朗沃尔不等式，无需罗宾斯-西格蒙德定理，恢复了两类方法的结果：对非凸目标，有 $\min_{s\leq t} \|\nabla F(w_s)\|^2 = o(t^{p-1})$；对凸目标，有 $\min_{s \leq t} F(w_s) - F_* = o(t^{p-1})$；对凸目标且 $β∈(0,1)$，有 $F(w_{τ\wedge t}) - F_* = o(t^{2γ/(1+γ) \cdot \max(p-1,-2p+1)-ε})$，其中 $τ:= \inf \{ t > 0 : F(w_t) = F_*\}$。此外，证明了常数动量 $β∈(0,1)$ 的 SHB 在 $γ=1$、步长 $α_t = Θ(t^{-p})$ 且 $p∈(\frac{1}{2},1)$ 时，以至少 $1-δ$ 的概率达到 $F(w_t) - F_* = O(t^{\max(p-1,-2p+1)} \log^2 \frac{t}{δ})$ 的收敛速率。
原文摘要 · Abstract (English)
We study the convergence rate for the last iterate of stochastic gradient descent (SGD) and stochastic heavy ball (SHB) in the parametric setting when the objective function $F$ is globally convex or non-convex whose gradient is $γ$-Hölder. Using only discrete Gronwall's inequality without Robbins-Siegmund theorem, we recover results for both SGD and SHB: $\min_{s\leq t} \|\nabla F(w_s)\|^2 = o(t^{p-1})$ for non-convex objectives and $F(w_{τ\wedge t}) - F_* = o(t^{2γ/(1+γ) \cdot \max(p-1,-2p+1)-ε})$ for $β\in (0, 1)$, $τ:= \inf \{ t > 0 : F(w_t) = F_*\}$, and $\min_{s \leq t} F(w_s) - F_* = o(t^{p-1})$ for convex objectives $F$ whose minimum is $F_*$. In addition, we proved that SHB with constant momentum parameter $β\in (0, 1)$ attains a convergence rate of $F(w_t) - F_* = O(t^{\max(p-1,-2p+1)} \log^2 \frac{t}δ)$ with probability at least $1-δ$ when $F$ is convex and $γ= 1$ and step size $α_t = Θ(t^{-p})$ with $p \in (\frac{1}{2}, 1)$.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。