自适应计算学习率，让训练更稳定且无需调参。
Taking the Road Less Scheduled with Adaptive Polyak Steps
- 用损失、梯度和当前参数动态计算学习率
- 在凸函数上达到1/√t的收敛速度
- 适合对超参数敏感的模型训练场景
Schedule-Free SGD 在无需预设训练时长的情况下实现了最优收敛速率，但仍需调优基础学习率。本文提出基于Polyak的自适应步长，仅依赖采样损失、梯度和当前迭代值即可实时计算学习率。首先提出一种使用每样本最优函数值的预言机变体，证明了凸Lipschitz目标下$O(1/\ ext{\sqrt{t}})$的任意时刻最后迭代率；随后引入保障版本，用已知下界替代未知最优值，在插值条件下收敛误差趋近于零。两种步长在无动量时退化为经典Polyak规则，统一了标准与无调度方法。语言建模实验（含预训练与蒸馏）表明，新方法在性能上匹配或超越调优后的基线，且对超参数更鲁棒。
原文摘要 · Abstract (English)
Schedule-Free SGD, proposed in [Defazio et al., 2024], achieves optimal convergence rates without requiring the training horizon in advance, by replacing learning rate schedules with a principled form of iterate averaging. However, the method still requires tuning a base learning rate whose optimal value depends on unknown problem constants. In this work, we continue down this road by deriving Polyak-type step sizes for Schedule-Free SGD and Adam that compute the learning rate at each iteration from the sampled loss, gradient, and current iterates alone. We first propose an oracle variant that uses per-sample optimal function values and prove an $O(1/\sqrt{t})$ anytime last-iterate rate for convex Lipschitz objectives. We then remove the oracle requirement with a safeguarded variant that replaces the unknown optimal values with any available lower bound, achieving the same rate up to a neighborhood that vanishes under interpolation. Both step sizes reduce to existing Polyak rules for standard SGD when momentum is set to zero, unifying standard and schedule-free Polyak methods. Numerical experiments on language modeling, including pretraining and distillation, show that the proposed methods match or surpass tuned Schedule-Free baselines while offering greater robustness to hyperparameter choices.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。