LightSAM自适应调整优化参数，让SAM不再依赖调参。
LightSAM: Parameter-Agnostic Sharpness-Aware Minimization
- 用自适应优化器替代SGD，动态调节扰动半径和学习率。
- 理论证明可在任意参数下收敛，实现真正无参优化。
- 适合需要稳定训练且不想调参的深度学习应用。
Sharpness-Aware Minimization (SAM) 通过权重扰动探索平坦极小值，提升模型泛化能力。尽管效果显著，但其引入额外超参数——扰动半径，导致对参数敏感。已有研究证明，该半径与学习率受问题相关参数约束才能保证收敛，限制了实际应用中的可调性。本文提出 LightSAM，通过自适应设定扰动半径与学习率，扩展 SAM 的适用范围。LightSAM 采用 AdaGrad-Norm、AdaGrad 及 Adam 三种主流自适应优化器替代 SGD 进行权重扰动与模型更新，降低对参数的敏感度。理论分析表明，在弱假设下，LightSAM 能在任意扰动半径与学习率设置下理想收敛，实现参数无关性。初步实验在多个深度学习任务上验证了其有效性。
原文摘要 · Abstract (English)
Sharpness-Aware Minimization (SAM) optimizer enhances the generalization ability of the machine learning model by exploring the flat minima landscape through weight perturbations. Despite its empirical success, SAM introduces an additional hyper-parameter, the perturbation radius, which causes the sensitivity of SAM to it. Moreover, it has been proved that the perturbation radius and learning rate of SAM are constrained by problem-dependent parameters to guarantee convergence. These limitations indicate the requirement of parameter-tuning in practical applications. In this paper, we propose the algorithm LightSAM which sets the perturbation radius and learning rate of SAM adaptively, thus extending the application scope of SAM. LightSAM employs three popular adaptive optimizers, including AdaGrad-Norm, AdaGrad and Adam, to replace the SGD optimizer for weight perturbation and model updating, reducing sensitivity to parameters. Theoretical results show that under weak assumptions, LightSAM could converge ideally with any choices of perturbation radius and learning rate, thus achieving parameter-agnostic. We conduct preliminary experiments on several deep learning tasks, which together with the theoretical findings validate the the effectiveness of LightSAM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。