为SAM优化器设计自适应步长,减少调参成本。
Adaptive Sharpness-Aware Minimization with a Polyak-type Step size: A Theory-Grounded Scheduler
- 基于Polyak步长思想,为SAM设计自适应学习率调度器。
- 理论证明在光滑条件下可线性收敛,且随机情形下逼近最优解。
- 实验显示性能媲美调优后的SAM,显著降低对学习率敏感度。
Sharpness-Aware Minimization (SAM) 已成为训练机器学习模型的有力且广泛应用的优化器。通过显式最小化损失曲面的尖锐度,SAM 通常能提升泛化能力并取得强劲的实证表现。然而,与大多数训练算法一样,SAM 及其变体对学习率的选择非常敏感,通常需通过大量超参数调优或预设调度器来确定。本文受近期关于随机 Polyak 步长在随机梯度下降(SGD)中有效性的启发,推导出适用于 SAM 式更新的 Polyak 调度器,分别在确定性和随机设定下提出新型自适应算法。在光滑设定下,我们证明了强凸目标函数下的线性收敛性,以及凸目标函数在确定性情况下的 $/mathcal{O}(1/T)$ 收敛速率。在随机设定下,我们建立了逼近最优解邻域的类似收敛保证。数值实验表明,所提出的 Polyak 调度器在性能上可媲美甚至优于精心调参的 SAM 基线,同时大幅减少了对学习率调优的需求。
原文摘要 · Abstract (English)
Sharpness-Aware Minimization (SAM) has established itself as a powerful and widely adopted optimizer for training machine learning models. By explicitly minimizing the sharpness of the loss landscape, SAM often improves generalization while delivering strong empirical performance. However, SAM and its variants, like most training algorithms, are sensitive to the choice of learning rate, which is typically selected through extensive hyperparameter tuning or predefined schedulers. In this work, motivated by recent advances on the effectiveness of stochastic Polyak step sizes for Stochastic Gradient Descent (SGD), we derive Polyak schedulers tailored to SAM-style updates, yielding novel adaptive algorithms in both deterministic and stochastic settings. In the smooth setting, we prove linear convergence for strongly convex objectives and an $\mathcal{O}(1/T)$ convergence rate for convex objectives in the deterministic case. In the stochastic setting, we establish analogous convergence guarantees up to a neighborhood of the optimum. Numerical experiments demonstrate that the proposed Polyak schedulers achieve performance comparable to or better than carefully tuned SAM baselines, while substantially reducing the need for learning-rate tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。