揭示随机梯度噪声如何让SAM随小批量提升泛化能力
Unveiling m-Sharpness Through the Structure of Stochastic Gradient Noise
- 用扩展SDE框架分析梯度噪声，发现扰动方差隐含正则化作用
- m越小，隐式正则强度越大，解释了SAM性能随微批次减小而提升
- 提出可并行的RW-SAM，模拟m-SAM优势且不依赖小批量训练
Sharpness-aware minimization (SAM) 是一种有效提升模型泛化能力的方法，但其内在机制尚未完全明晰。本文研究 m-sharpness 现象：当计算扰动的微批次大小降低时，SAM 性能单调提升，这对分布式训练至关重要却缺乏严谨解释。通过扩展的随机微分方程（SDE）框架分析随机梯度噪声（SGN），我们刻画了 n-SAM 和 m-SAM 等变体的动力学行为。分析表明，随机扰动会引入一种随 m 减小而增强的隐式方差型尖锐度正则化。基于此，我们提出 Reweighted SAM（RW-SAM），采用尖锐度加权采样以模拟 m-SAM 的泛化优势，同时保持可并行性。大量实验验证了理论与方法的有效性。代码已开源。
原文摘要 · Abstract (English)
Sharpness-aware minimization (SAM) has emerged as a highly effective technique to improve model generalization, but its underlying principles are not fully understood. We investigate m-sharpness, where SAM performance improves monotonically as the micro-batch size for computing perturbations decreases, a phenomenon critical for distributed training yet lacking rigorous explanation. We leverage an extended Stochastic Differential Equation (SDE) framework and analyze stochastic gradient noise (SGN) to characterize the dynamics of SAM variants, including n-SAM and m-SAM. Our analysis reveals that stochastic perturbations induce an implicit variance-based sharpness regularization whose strength increases as m decreases. Motivated by this insight, we propose Reweighted SAM (RW-SAM), which employs sharpness-weighted sampling to mimic the generalization benefits of m-SAM while remaining parallelizable. Comprehensive experiments validate our theory and method.Code is available at https://github.com/RitianLuo/RW-SAM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。