arXiv:2605.22644cs.LG2026-05

重新定义SGD动力学,揭示其非布朗运动的本质

Why SGD is not Brownian Motion: A New Perspective on Stochastic Dynamics

论文配图:Why SGD is not Brownian Motion: A New Perspective on Stochastic Dynamics
图 1 · 摘自论文原文
  • 将SGD视为采样引起的损失曲面波动下的确定性演化
  • 发现平坦方向上方差随时间增长,扩散系数与学习率成正比
  • 适用于理解神经网络优化中的收敛与发散机制

随机梯度下降(SGD)常被建模为朗之万过程,假设小批量噪声呈布朗运动。但这一近似依赖于连续时间极限和√η的噪声缩放,与有限学习率下的离散更新不一致。本文提出一种新框架:将SGD视为由小批量采样诱导的波动损失曲面中的确定性动力学。从离散更新出发,推导出参数分布的主方程,得到不同于标准朗之万形式的离散福克-普朗克方程,其在η²阶上存在差异。在此框架下分析损失函数临界点附近的动力学,发现行为可分解为均值海森矩阵的特征基,呈现定性不同的演化模式。特别是,在几乎平坦的方向上不存在稳态分布,方差随时间增长,对应沿谷底的有效扩散,扩散系数与学习率成正比。我们在计算机视觉和自然语言处理的神经网络模型上提供了实证支持,观察到受限模式与扩散模式之间的清晰定性区分。

原文摘要 · Abstract (English)

Stochastic Gradient Descent (SGD) is commonly modeled as a Langevin process, assuming that minibatch noise acts as Brownian motion. However, this approximation relies on a continuous-time limit and a sqrt(eta) noise scaling that does not match the discrete SGD update at finite learning rate. In this work, we propose an alternative formulation of SGD as deterministic dynamics in a fluctuating loss landscape induced by minibatch sampling. Starting directly from the discrete update, we derive a master equation for the parameter distribution and obtain a discrete Fokker--Planck equation that differs from the standard Langevin form at order eta^2. Using this framework, we analyze SGD dynamics near critical points of the loss. We show that the behavior decomposes along the eigenbasis of the mean Hessian into qualitatively distinct regimes. In particular, nearly-flat directions do not admit a stationary distribution: the variance grows over time, corresponding to effective diffusion along valleys with a coefficient proportional to the learning rate. We provide empirical evidence supporting these predictions on neural network models in computer vision and natural language processing, observing a clear qualitative separation between confined and diffusive modes.

优化器SGD动力学神经网络

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。