揭示非线性项如何影响梯度下降的稳定性，挑战传统线性分析。
On the Stability of Nonlinear Dynamics in GD and SGD: Beyond Quadratic Potentials
- 基于高阶导数推导出多维下梯度下降稳定振荡的精确条件。
- 发现单个批次不稳定时，随机梯度下降的期望仍可能发散。
- 证明所有批次线性稳定时，随机梯度下降的非线性动态在期望下稳定。
优化算法迭代过程中的动态稳定性决定了最终收敛到的极小值。例如，梯度下降（GD）的稳定解对应平坦极小值，被认为具有优良性质。以往研究常依赖线性化分析稳定性，但其是否能准确反映完整非线性行为尚不明确。近期工作表明，即使线性不稳定，梯度下降仍可能在步长衰减后稳定振荡并收敛，说明线性分析可能产生误导。本文显式研究非线性项的影响：首先在多变量情形下推导出梯度下降在极小值附近稳定振荡的精确条件，该条件依赖于高阶导数，推广了已有结果。将分析扩展至随机梯度下降（SGD），发现即使单个批次不稳定，非线性动态在期望上仍可能发散。这意味着稳定性可能由单个波动批次决定，而非平均效应，与线性分析假设相反。最后证明：若所有批次均线性稳定，则SGD的非线性动态在期望下稳定。
原文摘要 · Abstract (English)
The dynamical stability of the iterates during training plays a key role in determining the minima obtained by optimization algorithms. For example, stable solutions of gradient descent (GD) correspond to flat minima, which have been associated with favorable features. While prior work often relies on linearization to determine stability, it remains unclear whether linearized dynamics faithfully capture the full nonlinear behavior. Recent work has shown that GD may stably oscillate near a linearly unstable minimum and still converge once the step size decays, indicating that linear analysis can be misleading. In this work, we explicitly study the effect of nonlinear terms. Specifically, we derive an exact criterion for stable oscillations of GD near minima in the multivariate setting. Our condition depends on high-order derivatives, generalizing existing results. Extending the analysis to stochastic gradient descent (SGD), we show that nonlinear dynamics can diverge in expectation even if a single batch is unstable. This implies that stability can be dictated by a single batch that oscillates unstably, rather than an average effect, as linear analysis suggests. Finally, we prove that if all batches are linearly stable, the nonlinear dynamics of SGD are stable in expectation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。