通过增大批次或衰减学习率，使GSAM找到更平坦的极小值。
Convergence of Sharpness-Aware Minimization Algorithms using Increasing Batch Size and Decaying Learning Rate
- 在训练中逐步增大批次或衰减学习率，提升模型泛化能力。
- 实验表明，动态调整参数可找到比固定参数更平坦的局部极小值。
- 适合关注模型优化与泛化性能的研究者参考。
Sharpness-aware minimization (SAM) 及其变体 gap guided SAM (GSAM) 通过寻找经验损失函数的平坦局部极小值，显著提升了深度神经网络的泛化能力。已有理论和实证研究表明，增大批次大小或衰减学习率有助于避开尖锐局部极小值。本文研究了采用递增批次大小或衰减学习率（如余弦退火、线性衰减）的 GSAM 算法，并从理论上证明其收敛性。此外，数值实验对比了带与不带递增批次大小的 SAM (GSAM)，结果表明：使用递增批次大小或衰减学习率能获得比恒定批次与学习率更平坦的局部极小值。
原文摘要 · Abstract (English)
The sharpness-aware minimization (SAM) algorithm and its variants, including gap guided SAM (GSAM), have been successful at improving the generalization capability of deep neural network models by finding flat local minima of the empirical loss in training. Meanwhile, it has been shown theoretically and practically that increasing the batch size or decaying the learning rate avoids sharp local minima of the empirical loss. In this paper, we consider the GSAM algorithm with increasing batch sizes or decaying learning rates, such as cosine annealing or linear learning rate, and theoretically show its convergence. Moreover, we numerically compare SAM (GSAM) with and without an increasing batch size and conclude that using an increasing batch size or decaying learning rate finds flatter local minima than using a constant batch size and learning rate.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。