改进了基于路径的自适应学习率方法，让SGD和Adam训练更稳定高效。
Cumulative Learning Rate Adaptation: Revisiting Path-Based Schedules for SGD and Adam
- 通过累积路径长度动态调整学习率，响应损失曲面变化
- 发现原方法在Adam上存在机制矛盾，提出修正版本适配其更新逻辑
- 实验表明该方法在多种场景下提升模型收敛性，适合追求稳定训练的研究者
学习率是深度学习中至关重要的超参数，其最优值依赖于具体任务且可能随训练过程变化。本文重新研究了2017年提出的基于累积路径的自适应学习率机制，该方法根据观测路径长度（以归一化梯度步长的时间折扣和计算）与随机游走预期长度之间的差异来调整学习率。尽管该方法直觉清晰，我们发现其在Adam优化器上的适应机制存在概念不一致问题，因Adam内部存在预条件化。为此，我们提出一种修正方案，使其更准确反映Adam的更新动态。为评估在线学习率自适应的实际价值，我们在SGD和Adam上对比了是否采用累积适应策略，并与近期另一方法进行比较。结果旨在阐明此类自适应策略在何种条件下能带来实际收益。
原文摘要 · Abstract (English)
The learning rate is a crucial hyperparameter in deep learning, with its ideal value depending on the problem and potentially changing during training. In this paper, we investigate the practical utility of adaptive learning rate mechanisms that adjust step sizes dynamically in response to the loss landscape. We revisit a cumulative path-based adaptation scheme proposed in 2017, which adjusts the learning rate based on the discrepancy between the observed path length, computed as a time-discounted sum of normalized gradient steps, and the expected length of a random walk. While the original approach offers a compelling intuition, we show that its adaptation mechanism for Adam is conceptually inconsistent due to the optimizer's internal preconditioning. We propose a corrected variant that better reflects Adam's update dynamics. To assess the practical value of online learning rate adaptation, we benchmark SGD and Adam, with and without cumulative adaptation, and compare them to a recent alternative method. Our results aim to clarify when and why such adaptive strategies offer practical benefits.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。