提出随机方向的随机梯度下降法，提升优化灵活性与收敛性。
A stochastic gradient descent algorithm with random search directions
- 用随机方向代替坐标轴方向进行梯度下降
- 证明了递减步长下几乎必然收敛，给出非渐近Lp收敛率
- 分析了方向分布对渐近协方差的影响，适合理论优化研究者
随机坐标下降法通过固定大部分坐标，仅优化剩余坐标，是高效优化方法，但通常局限于R^d的标准基向量。本文提出一类新型随机梯度下降算法，采用更一般的随机向量作为搜索方向，并基于梯度估计的方向导数进行更新。建立了递减步长下的几乎必然收敛性，进一步研究了其中心极限定理，特别分析了搜索分布对渐近协方差矩阵的影响。同时给出了非渐近Lp收敛速率。
原文摘要 · Abstract (English)
Stochastic coordinate descent algorithms are efficient methods in which each iterate is obtained by fixing most coordinates at their values from the current iteration, and approximately minimizing the objective with respect to the remaining coordinates. However, this approach is usually restricted to canonical basis vectors of $\mathbb{R}^d$. In this paper, we develop a new class of stochastic gradient descent algorithms with random search directions which uses the directional derivative of the gradient estimate following more general random vectors. We establish the almost sure convergence of these algorithms with decreasing step. We further investigate their central limit theorem and pay particular attention to analyze the impact of the search distributions on the asymptotic covariance matrix. We also provide non-asymptotic $\mathbb{L}^p$ rates of convergence.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。