arXiv:2608.03001math.OCcs.LG2026-08

突破传统噪声假设,证明随机优化可避开鞍点

Stochastic Saddle Avoidance Beyond Unit Excitation and Smoothness: A Pathwise Lyapunov-Perron Framework

  • 提出路径依赖的李雅普诺夫-佩龙框架,无需单位激励假设
  • 在非光滑、低维噪声等复杂场景下仍能保证鞍点规避
  • 适用于梯度下降、重排采样等常见优化算法,适合研究者参考

单位激励(UE)是随机鞍点规避中的常见假设:期望下随机误差在每个方向上均有正分量。该假设虽能直接排除收敛至严格鞍点,但过度简化了真实噪声结构,不适用于过参数化或插值模型中噪声趋于零的情形,也不符合有限求和问题中噪声位于低维数据相关子空间的情况。本文提出一个无需UE的抽象几乎必然规避定理,以可验证的路径相关条件替代原要求。这些条件在标准独立同分布采样下由局部光滑性和有限矩假设推出,或在无放回采样下的有限求和结构中成立。由于随机采样映射通常无固定点,经典中心-稳定流形分析不再适用。本文采用路径相关变量变换与路径式李雅普诺夫-佩龙策略进行证明。应用方面,我们获得了随机镜面下降(包括SGD)和随机重排的严格鞍点规避结果;对非光滑复合目标,还给出了近端型随机梯度法的规避结论。结合合适的迭代收敛保证,可进一步证明收敛至原始目标函数的局部极小值。

原文摘要 · Abstract (English)

Unit excitation (UE) is a common assumption in stochastic saddle avoidance: the stochastic error must have a uniformly positive component along every direction, in expectation. This condition gives a direct way to rule out convergence to strict saddles, but it also oversimplifies the actual noise structure, and does not match many stochastic optimization regimes. In overparameterized or interpolation models, the noise may vanish near stationarity. In finite-sum problems, the stochastic gradient noise may lie in a low-dimensional, data-dependent subspace. In these (common) scenarios, UE is naturally not satisfied. In this paper, we prove an abstract almost sure avoidance theorem for stochastic recursions without UE. The theorem replaces UE-type requirements by verifiable pathwise conditions. In applications, these conditions follow, e.g., from local smoothness and finite-moment assumptions under standard i.i.d. sampling, or from the finite-sum structure under without-replacement sampling. Since the stochastically sampled maps generally do not share a fixed point, the celebrated center-stable manifold argument used in deterministic analyses is not directly applicable. Instead, we use a path-dependent change of variables together with a pathwise Lyapunov--Perron-based proof strategy. As applications, we obtain strict saddle avoidance for stochastic mirror descent (including SGD) and for random reshuffling. For nonsmooth composite objectives, we prove avoidance results for a proximal-type stochastic gradient method. Combining these insights with suitable iterate convergence guarantees, this allows establishing convergence to local minimizers of the original objective function.

随机优化鞍点规避非光滑优化收敛性分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。