发现平坦极小值未必泛化好,算法设计需更谨慎。
Flat Minima and Generalization: Insights from Stochastic Convex Optimization
- 在凸优化中用平滑性分析极小值平坦度与泛化关系。
- 即使算法找到平坦解,仍可能有Ω(1)的总体风险。
- 两类尖锐感知算法(SA-GD、SAM)也存在泛化失效问题。
理解学习算法的泛化行为是学习理论的核心目标。近年来一种流行解释认为,算法之所以有效,是因为收敛到平坦极小值,而平坦极小值与更好的泛化性能相关。本文研究了在非负、β-光滑的随机凸优化经典设定下,平坦极小值与泛化之间的联系。首先发现,即便在此基础且广泛研究的设置中,平坦的经验极小值也可能导致平凡的Ω(1)总体风险,而尖锐极小值却能实现最优泛化。进一步证明,两种由Foret等人(2021)提出的自然“尖锐感知”算法——尖锐感知梯度下降(SA-GD)和尖锐感知最小化(SAM)——同样表现出此类差泛化行为。对于在预定义邻域内进行最大损失梯度步的SA-GD,我们证明其虽以快速率收敛至平坦极小值,但解的总体风险仍可达Ω(1),表明算法寻得的平坦极小值也可能泛化不佳。对于基于归一化上升步的计算高效近似算法SAM,我们发现其虽可最小化经验损失,却可能收敛至尖锐极小值,并产生Ω(1)的总体风险。最后,我们利用算法稳定性技术为两者建立了总体风险上界。
原文摘要 · Abstract (English)
Understanding the generalization behavior of learning algorithms is a central goal of learning theory. A recently emerging explanation is that learning algorithms are successful in practice because they converge to flat minima, which have been consistently associated with improved generalization performance. In this work, we study the link between flat minima and generalization in the canonical setting of stochastic convex optimization with a non-negative, $β$-smooth objective. Our first finding is that, even in this fundamental and well-studied setting, flat empirical minima may incur trivial $Ω(1)$ population risk while sharp minima generalizes optimally. Then, we show that this poor generalization behavior extends to two natural ''sharpness-aware'' algorithms originally proposed by Foret et al. (2021), designed to bias optimization toward flat solutions: Sharpness-Aware Gradient Descent (SA-GD) and Sharpness-Aware Minimization (SAM). For SA-GD, which performs gradient steps on the maximal loss in a predefined neighborhood, we prove that while it successfully converges to a flat minimum at a fast rate, the population risk of the solution can still be as large as $Ω(1)$, indicating that even flat minima found algorithmically using a sharpness-aware gradient method might generalize poorly. For SAM, a computationally efficient approximation of SA-GD based on normalized ascent steps, we show that although it minimizes the empirical loss, it may converge to a sharp minimum and also incur population risk $Ω(1)$. Finally, we establish population risk upper bounds for both SA-GD and SAM using algorithmic stability techniques.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。