arXiv:2509.21818cs.LGmath.OC2025-09

SAM优化器可能陷入虚假极小值,影响模型性能

Sharpness-Aware Minimization Can Hallucinate Minimizers

  • SAM通过扰动梯度方向寻找平坦极小值
  • 在特定扰动距离下会停滞于非真实极小点
  • 预热SGD可有效避免该问题,降低参数敏感性

Sharpness-Aware Minimization (SAM) 广泛用于寻找平坦极小值,通常与更好的泛化性能相关。其标准实现中,使用沿归一化梯度方向偏移距离 ρ 后的点计算损失梯度来更新当前迭代点。我们发现,当 ρ 取某些值时,SAM 可能在原始梯度非零的情况下停滞于扰动点梯度为零的位置,这些点并非原损失函数的驻点,称为‘幻觉极小值’。我们在简单非凸景观条件下(如存在局部极小和局部极大)证明了这类点的存在性,并给出了迭代收敛到它们的充分条件。在神经网络训练中验证了该失效模式,发现其与大 ρ 下 SAM 性能下降现象一致。最后,我们提出一种实用防护策略:在启用 SAM 前进行短周期 SGD 预热,可缓解该问题并降低对 ρ 的敏感性。

原文摘要 · Abstract (English)

Sharpness-Aware Minimization (SAM) is widely used to seek flatter minima -- often linked to better generalization. In its standard implementation, SAM updates the current iterate using the loss gradient evaluated at a point perturbed by distance $ρ$ along the normalized gradient direction. We show that, for some choices of $ρ$, SAM can stall at points where this shifted (perturbed-point) gradient vanishes despite a nonzero original gradient, and therefore, they are not stationary points of the original loss. We call these points hallucinated minimizers, prove their existence under simple nonconvex landscape conditions (e.g., the presence of a local minimizer and a local maximizer), and establish sufficient conditions for local convergence of the SAM iterates to them. We corroborate this failure mode in neural network training and observe that it aligns with SAM's performance degradation often seen at large $ρ$. Finally, as a practical safeguard, we find that a short initial SGD warm-start before enabling SAM mitigates this failure mode and reduces sensitivity to the choice of $ρ$.

优化器SAM泛化神经网络

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。