SGD不主动追求平坦，而是选择噪声均衡时的低梯度波动解。
Does SGD Seek Flatness or Sharpness? An Exactly Solvable Model
- 通过可解析模型揭示SGD本质偏好：最小化梯度波动而非直接追求平坦
- 标签噪声各向同性时收敛到平坦解，各向异性则趋向任意尖锐解
- 适用于理解优化行为的理论分析，适合研究深度学习泛化机制者
大量理论与实证研究认为神经网络损失曲面的平坦性与性能相关。然而，关于SGD在训练中更倾向平坦或尖锐解存在相反证据。本文通过构建一个可精确求解的模型，部分但因果地澄清了SGD的平坦性偏好：该模型中SGD并无预先的平坦偏好，仅倾向于最小化梯度波动。关键发现是，收敛后的尖锐程度完全由数据分布决定；当标签噪声在所有输出维度上各向同性时，模型偏好平坦极小值；若噪声各向异性，则模型偏好尖锐解，且可收敛至任意尖锐程度，具体取决于标签噪声谱的失衡程度。该结论在MLP、RNN和Transformer等多种架构的受控设置中均得到验证。
原文摘要 · Abstract (English)
A large body of theory and empirical work hypothesizes a connection between the flatness of a neural network's loss landscape during training and its performance. However, there have been conceptually opposite pieces of evidence regarding when SGD prefers flatter or sharper solutions during training. In this work, we partially but causally clarify the flatness-seeking behavior of SGD by identifying and exactly solving an analytically solvable model that exhibits both flattening and sharpening behavior during training. In this model, the SGD training has no \textit{a priori} preference for flatness, but only a preference for minimal gradient fluctuations. This leads to the insight that, at least within this model, it is data distribution that uniquely determines the sharpness at convergence, and that a flat minimum is preferred if and only if the noise in the labels is isotropic across all output dimensions. When the noise in the labels is anisotropic, the model instead prefers sharpness and can converge to an arbitrarily sharp solution, depending on the imbalance in the noise in the labels spectrum. We reproduce this key insight in controlled settings with different model architectures such as MLP, RNN, and transformers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。