arXiv:2603.08290cs.LGcs.AI2026-03中稿 · ICLR被引 1

SAM在深层网络中会优先关注次要特征,导致训练初期依赖不重要特征。

Minor First, Major Last: A Depth-Induced Implicit Bias of Sharpness-Aware Minimization

  • 通过梯度归一化放大小特征,使模型初期依赖次要坐标
  • 深层网络下,$oldsymbol{ heta}$ 会从任意方向收敛至零或标准基向量
  • 揭示了传统极限分析的不足,适合研究优化器偏见的学者

我们研究了在 $L$ 层线性对角网络上训练线性可分二分类任务时,尖锐感知最小化(SAM)的隐式偏差。对于线性模型($L=1$),$oldsymbol{ heta}$-和 $oldsymbol{ heta}_2$-SAM 均恢复 $oldsymbol{ heta}_2$ 最大间隔分类器,与梯度下降(GD)一致。然而,当深度 $L = 2$ 时,行为发生剧烈变化——即使在单样本数据集上也如此。对于 $oldsymbol{ heta}_oldsymbol{ heta}$-SAM,极限方向强烈依赖于初始化,可能收敛至 $oldsymbol{0}$ 或任一标准基向量;而 GD 的极限则始终对齐主导数据坐标。对于 $oldsymbol{ heta}_2$-SAM,尽管其极限方向匹配 $oldsymbol{ heta}_1$ 最大间隔解,但其有限时间动态呈现‘顺序特征放大’现象:预测器初期依赖次要坐标,随训练推进或初始化增强逐渐转向主要坐标。理论分析表明,该现象源于扰动中应用的 $oldsymbol{ heta}_2$-SAM 梯度归一化因子,放大了次要坐标并允许主要坐标后期主导。合成与真实数据实验验证了上述发现。

原文摘要 · Abstract (English)

We study the implicit bias of Sharpness-Aware Minimization (SAM) when training $L$-layer linear diagonal networks on linearly separable binary classification. For linear models ($L=1$), both $\ell_\infty$- and $\ell_2$-SAM recover the $\ell_2$ max-margin classifier, matching gradient descent (GD). However, for depth $L = 2$, the behavior changes drastically -- even on a single-example dataset. For $\ell_\infty$-SAM, the limit direction depends critically on initialization and can converge to $\mathbf{0}$ or to any standard basis vector, in stark contrast to GD, whose limit aligns with the basis vector of the dominant data coordinate. For $\ell_2$-SAM, we show that although its limit direction matches the $\ell_1$ max-margin solution as in the case of GD, its finite-time dynamics exhibit a phenomenon we call "sequential feature amplification", in which the predictor initially relies on minor coordinates and gradually shifts to larger ones as training proceeds or initialization increases. Our theoretical analysis attributes this phenomenon to $\ell_2$-SAM's gradient normalization factor applied in its perturbation, which amplifies minor coordinates early and allows major ones to dominate later, giving a concrete example where infinite-time implicit-bias analyses are insufficient. Synthetic and real-data experiments corroborate our findings.

优化器偏差深度学习隐式偏好

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。