arXiv:2608.03197cs.LG2026-08

SAM通过控制扰动半径,让模型自动找更平坦的解,提升泛化能力。

On the Implicit Flatness Bias of Sharpness-Aware Minimization: A Linear Stability Analysis with Quantitative Hyperparameter Bounds

  • 基于线性稳定性分析,揭示了SAM隐含的平坦极小值偏好机制。
  • 发现更小的批次、更大的学习率或更大的扰动半径会引导模型走向更平坦的解。
  • 提出TLC-SAM,动态调整扰动半径,进一步降低损失曲面最陡方向的曲率。

Sharpness-Aware Minimization (SAM) 通过寻找对局部对抗扰动鲁棒的参数来提升泛化性能,但其隐含偏向平坦极小值的定量机制尚不清晰。特别是扰动半径 $ρ$ 通常被视为独立超参数,而它实际上定义了衡量尖锐度的邻域范围。本文通过在插值极小值附近进行线性稳定性分析,假设梯度噪声对齐与局部线性化,在固定其他条件时,证明所有线性稳定极小值满足 $λ_{ ext{max}} \leq \sqrt[3]{bΓ/(2ρη^2)}$,其中 $λ_{\text{max}}$ 为海森矩阵最大特征值,$b$ 为批次大小,$η$ 为学习率,$Γ$ 为梯度范数上界。该边界定量刻画了SAM的平坦偏倚:在其他量固定时,较小批次、较大学习率或较大半径会使线性稳定解趋于更平坦。同时揭示必要权衡:$ρ$ 需足够大以促进平坦性,又需保持局部性以保证近似有效且训练稳定。在CIFAR-100上使用ResNet-18和VGG-19的900个模型控制实验中,增加 $ρ$ 均显著降低最大海森特征值。最后,提出泰勒局部性可控的SAM(TLC-SAM),依据观测泰勒逼近误差动态调整 $ρ$,相比固定半径的SAM进一步降低顶层海森特征值。研究提供量化超参数边界与稳定性-局部性视角,用于分析与设计SAM变体。

原文摘要 · Abstract (English)

Sharpness-Aware Minimization (SAM) improves generalization by seeking parameters whose loss is robust to local adversarial perturbations, but the quantitative mechanism underlying its implicit bias toward flat minima remains unclear. In particular, the perturbation radius $ρ$ is typically treated as an isolated tuning parameter, despite defining the neighborhood in which SAM measures sharpness. We analyze mini-batch SAM near an interpolating minimum through linear stability. Under local linearization and gradient-noise alignment assumptions, we prove that every linearly stable minimum satisfies $λ_{\max}\leq\sqrt[3]{bΓ/(2ρη^2)}$, where $λ_{\max}$ is the largest Hessian eigenvalue, $b$ is the batch size, $η$ is the learning rate, and $Γ$ bounds the gradient norm. The bound quantitatively characterizes SAM's implicit flatness bias: holding the other quantities fixed, a smaller batch size, a larger learning rate, or a larger radius restricts linearly stable SAM to flatter minima. It also exposes a necessary trade-off: $ρ$ should be large enough to promote flatness, yet remain local enough to preserve the approximation and stable training. We validate this prediction in a controlled study of 900 models on CIFAR-100 with ResNet-18 and VGG-19, where increasing $ρ$ is consistently associated with a smaller largest Hessian eigenvalue across batch-size and learning-rate settings. Finally, we instantiate the analysis in Taylor-Locality Controlled SAM (TLC-SAM), which adjusts $ρ$ using the observed Taylor-approximation error and further reduces the top Hessian eigenvalue relative to fixed-radius SAM. Our results provide quantitative hyperparameter bounds and a stability--locality perspective for analyzing and designing SAM variants.

优化算法平坦极小值神经网络训练超参分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。