提出一种受最速降线启发的学习率调度法,实测在多种模型上表现优异。
BrachistoneLR: A Brachistochrone-Inspired Learning-Rate Schedule and a Controlled Benchmark of Scheduling Policies
- 用最速降线原理设计学习率变化曲线,使速率在最后一轮达到最低值
- 在CIFAR-10上比恒定学习率提升2.5个百分点,优于常见调度策略
- 无需设定预热、重启或衰减参数,适合追求稳定性能的训练场景
学习率调度对深度网络训练至关重要,但现有策略多为启发式,且比较研究因架构、数据集和训练预算差异难以解读。本文提出BrachistoneLR,将最速降线的垂直坐标映射到学习率区间,实现从峰值到底值的平滑下降。分析表明其本质为周期设为E-1的余弦退火(标准实现中周期参数比总轮数少1),使学习率在最后训练轮达到最低,该差异随轮数平方衰减,属短时效应。在三个图像分类数据集(MNIST、Fashion-MNIST、CIFAR-10)和四种网络结构(全连接、卷积、循环、残差)上,固定优化器、数据管道与评估协议,仅改变调度策略进行72次实验。平滑下降型调度显著优于恒定率和日历衰减,难度越高优势越大,在CIFAR-10上平均提升达2.5分。在领先组中,BrachistoneLR、余弦退火与预热-余弦调度差距极小(精度差≤0.06,均值排名差≤0.17),单种子无法区分。BrachistoneLR在残差网络上最优,且在CIFAR-10上取得最高平均准确率,且无需设置里程碑、衰减因子、预热长度或重启周期。结论:调度形状比参数化更重要,是否采用平滑调度比具体选择哪种更关键,终端速率差异仅在短周期内有意义。
原文摘要 · Abstract (English)
The learning-rate schedule is a consequential choice in training deep networks, yet the policies in common use are heuristic, and published comparisons are hard to read, because architecture, dataset, and budget tend to vary alongside the schedule. We study BrachistoneLR, a schedule built by mapping the vertical coordinate of the brachistochrone, the curve of fastest descent under gravity, onto the range between a peak and a floor rate. Expanding the definition shows it to be cosine annealing with the half-period set to E - 1 instead of E, the configuration a standard implementation gives when its period argument is one less than the number of epochs. The rate therefore reaches its floor at the last epoch trained rather than one epoch later, and we show this difference decays as E^-2, making it a short-horizon effect. We then benchmark six schedules over 72 runs on three image classification datasets (MNIST, Fashion-MNIST, CIFAR-10) and four architecture families (fully connected, convolutional, recurrent, residual), fixing the optimizer, data pipeline, and evaluation protocol so that only the schedule varies. Schedules that fall smoothly from peak to floor beat the constant rate and calendar-based decay by margins that grow with task difficulty, reaching 2.5 points of dataset mean on CIFAR-10. Within that leading group, BrachistoneLR, cosine annealing, and warmup-cosine lie within 0.06 accuracy points and 0.17 of a mean rank, which one seed per configuration cannot separate. BrachistoneLR is best on both residual networks and has the highest CIFAR-10 mean, and it sets no milestones, decay factor, warmup length, or restart period. We conclude that the shape of a schedule matters more than its parameterization, that the choice of whether to use a smooth schedule matters more than the choice among them, and that the terminal-rate distinction is worth attention only over short horizons.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。