揭示了不同任务难度下最优学习率调度的规律，指导高效训练。
Optimal Learning-Rate Schedules under Functional Scaling Laws: Power Decay and Warmup-Stable-Decay
- 基于函数尺度律框架，推导出任务难度决定最优学习率变化形式。
- 简单任务用幂律衰减，复杂任务采用预热-稳定-衰减结构，提升训练效率。
- 理论证明可实现最优收敛速度，适合深度学习模型调参与算法设计者。
本文研究在Li等（2025）提出的函数尺度律（FSL）框架下最优学习率调度（LRS）的性质。该框架能准确描述线性回归和大语言模型（LLM）预训练中的损失动态。损失演化由两个指数控制：信号学习速率的源指数 $s>0$ 与噪声遗忘速率的容量指数 $β>1$。针对固定训练步数 $N$，我们推导出最优学习率调度，并发现明显的相变现象：当 $s \ge 1 - 1/β$（易任务），最优调度为幂律衰减至零，形式为 $η^*(z) = η_{\mathrm{peak}}(1 - z/N)^{2β-1}$，峰值学习率满足 $η_{\mathrm{peak}} \eqsim N^{-ν}$，其中 $ν= ν(s,β)$ 为显式定义的指数；当 $s < 1 - 1/β$（难任务），最优调度呈现预热-稳定-衰减（WSD）结构，在大部分训练期间维持最大允许学习率，仅末期衰减，且衰减阶段占比趋于零。我们进一步分析了仅调整峰值的学习率固定形状调度，揭示其优势与内在局限，为余弦、线性衰减等常用策略提供理论评估。最后，将幂律衰减调度应用于核回归的一次遍历随机梯度下降（SGD），证明最终迭代点达到精确的极小极大最优率，消除了以往分析中存在的对数次优性。数值实验验证了理论预测。
原文摘要 · Abstract (English)
We study optimal learning-rate schedules (LRSs) under the functional scaling law (FSL) framework introduced in Li et al. (2025), which accurately models the loss dynamics of both linear regression and large language model (LLM) pre-training. Within FSL, loss dynamics are governed by two exponents: a source exponent $s>0$ controlling the rate of signal learning, and a capacity exponent $β>1$ determining the rate of noise forgetting. Focusing on a fixed training horizon $N$, we derive the optimal LRSs and reveal a sharp phase transition. In the easy-task regime $s \ge 1 - 1/β$, the optimal schedule follows a power decay to zero, $η^*(z) = η_{\mathrm{peak}}(1 - z/N)^{2β- 1}$, where the peak learning rate scales as $η_{\mathrm{peak}} \eqsim N^{-ν}$ for an explicit exponent $ν= ν(s,β)$. In contrast, in the hard-task regime $s < 1 - 1/β$, the optimal LRS exhibits a warmup-stable-decay (WSD) (Hu et al. (2024)) structure: it maintains the largest admissible learning rate for most of training and decays only near the end, with the decay phase occupying a vanishing fraction of the horizon. We further analyze optimal shape-fixed schedules, where only the peak learning rate is tuned -- a strategy widely adopted in practiceand characterize their strengths and intrinsic limitations. This yields a principled evaluation of commonly used schedules such as cosine and linear decay. Finally, we apply the power-decay LRS to one-pass stochastic gradient descent (SGD) for kernel regression and show the last iterate attains the exact minimax-optimal rate, eliminating the logarithmic suboptimality present in prior analyses. Numerical experiments corroborate our theoretical predictions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。