揭示SGD在临界点附近的收敛、滞留与逃逸机制。
Convergence, Sticking and Escape: Stochastic Dynamics Near Critical Points in SGD
- 分析一维场景下噪声特性对SGD路径的影响
- 证明初始点远离极值时能稳定抵达局部极小
- 解释为何尖锐极大值附近会短暂滞留但最终逃逸
我们研究了一维景观中随机梯度下降（SGD）的收敛特性和逃逸动态，分别考虑了无限方差和有限方差噪声。在噪声分布满足一定条件下，我们证明了除非初始点过于接近局部极大值，否则SGD将可靠地收敛至同一带域内的局部最小值。当初始点靠近‘尖锐’极大值时，我们发现SGD会在此邻域长时间滞留，但不会永久停留，并给出了其到达两侧邻近极小值的概率估计。整体结果揭示了SGD在局部极大值与极小值之间过渡行为的复杂性，受噪声特征和函数几何结构共同影响。
原文摘要 · Abstract (English)
We study the convergence properties and escape dynamics of Stochastic Gradient Descent (SGD) in one-dimensional landscapes, separately considering infinite- and finite-variance noise. Our main focus is to identify the time scales on which SGD reliably moves from an initial point to the local minimum in the same ''basin''. Under suitable conditions on the noise distribution, we prove that SGD converges to the basin's minimum unless the initial point lies too close to a local maximum. In that near-maximum scenario, we show that SGD can linger for a long time in its neighborhood. For initial points near a ''sharp'' maximum, we show that SGD does not remain stuck there, and we provide results to estimate the probability that it will reach each of the two neighboring minima. Overall, our findings present a nuanced view of SGD's transitions between local maxima and minima, influenced by both noise characteristics and the underlying function geometry.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。