基于梯度波动自动调节学习率，无需调参且效果稳定。
Explainable Learning Rate Regimes for Stochastic Optimization
- 根据随机梯度范数动态调整学习率，实现自动调节。
- 在SGD、SGDM、SIGNSGD上均表现高效、鲁棒且可扩展。
- 无需额外调参，适合追求简洁高效的训练场景。
现代机器学习通常使用随机梯度下降（SGD）训练，其性能高度依赖于学习率（LR）随时间的调整与衰减策略。然而现有学习率方案往往复杂，或需手动调节一个或多个额外超参数，实际应用中面临巨大的计算开销、时间和能耗瓶颈。本文以自然直接的方式，阐明了学习率应仅根据随机梯度的内在变化自动更新。提出一种可解释的学习率策略，利用随机二阶算法实现，其行为模式类似启发式方法，但实现简单且无需任何参数调优。该策略规定：当随机梯度范数减小时，学习率应增加；反之则减小。所提学习率机制在多种经典随机优化算法（包括SGD、SGDM和SIGNSGD）上展现出高效性、鲁棒性和可扩展性，适用于各类机器学习任务。
原文摘要 · Abstract (English)
Modern machine learning is trained by stochastic gradient descent (SGD), whose performance critically depends on how the learning rate (LR) is adjusted and decreased over time. Yet existing LR regimes may be intricate, or need to tune one or more additional hyper-parameters manually whose bottlenecks include huge computational expenditure, time and power in practice. This work, in a natural and direct manner, clarifies how LR should be updated automatically only according to the intrinsic variation of stochastic gradients. An explainable LR regime by leveraging stochastic second-order algorithms is developed, behaving a similar pattern to heuristic algorithms but implemented simply without any parameter tuning requirement, where it is of an automatic procedure that LR should increase (decrease) as the norm of stochastic gradients decreases (increases). The resulting LR regime shows its efficiency, robustness, and scalability in different classical stochastic algorithms, containing SGD, SGDM, and SIGNSGD, on machine learning tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。