arXiv:2409.13173cs.CVcs.LG2024-09被引 2

改进SAM算法,找到更平坦的最小值,提升模型泛化能力。

Bilateral Sharpness-Aware Minimization for Flatter Minima

  • 结合最大尖锐度与最小尖锐度,构建更优平坦性指标
  • 在分类、姿态估计等任务上显著优于原版SAM
  • 适合追求高泛化性能的深度学习研究者

Sharpness-Aware Minimization (SAM) 通过降低最大尖锐度(MaxS)来增强泛化能力。尽管实践效果良好,但我们实证发现,SAM背后的最大尖锐度存在‘平坦性指示问题’(FIP),仅考虑梯度上升方向的平坦性,导致后续优化区域不够平坦。更好的平坦性指示器(FI)可带来更强的泛化能力。由于SAM本质上是贪婪搜索方法,本文提出利用当前权重邻域内训练损失与最小损失的差异,定义为最小尖锐度(MinS)。将MaxS与MinS融合,构建出更优的平坦性指示器,能更好地指示优化过程中的平坦方向。特别地,将该指标与SAM结合,提出双边SAM(BSAM),可找到比SAM更平坦的最小值。理论分析证明了BSAM收敛至局部极小点。大量实验表明,相比原始SAM,BSAM在分类、迁移学习、人体姿态估计和网络量化等多个任务中均表现出更优的泛化性能与鲁棒性。代码已公开于:https://github.com/ajiaaa/BSAM。

原文摘要 · Abstract (English)

Sharpness-Aware Minimization (SAM) enhances generalization by reducing a Max-Sharpness (MaxS). Despite the practical success, we empirically found that the MAxS behind SAM's generalization enhancements face the "Flatness Indicator Problem" (FIP), where SAM only considers the flatness in the direction of gradient ascent, resulting in a next minimization region that is not sufficiently flat. A better Flatness Indicator (FI) would bring a better generalization of neural networks. Because SAM is a greedy search method in nature. In this paper, we propose to utilize the difference between the training loss and the minimum loss over the neighborhood surrounding the current weight, which we denote as Min-Sharpness (MinS). By merging MaxS and MinS, we created a better FI that indicates a flatter direction during the optimization. Specially, we combine this FI with SAM into the proposed Bilateral SAM (BSAM) which finds a more flatter minimum than that of SAM. The theoretical analysis proves that BSAM converges to local minima. Extensive experiments demonstrate that BSAM offers superior generalization performance and robustness compared to vanilla SAM across various tasks, i.e., classification, transfer learning, human pose estimation, and network quantization. Code is publicly available at: https://github.com/ajiaaa/BSAM.

优化算法泛化能力神经网络

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。