AutoSGD自动调节学习率，让SGD训练更省心
AutoSGD: Automatic Learning Rate Selection for Stochastic Gradient Descent
- 根据当前梯度变化自动决定学习率增减
- 在多种机器学习任务中表现优于手动调参
- 理论保证收敛性，适合不想调参的研究者
学习率是随机梯度下降（SGD）的重要调参参数，对性能影响显著。然而，在所有迭代中选择合适的学习率调度通常需要大量人工调优。为此，我们提出AutoSGD：一种能自动判断当前迭代是否应提高或降低学习率，并采取相应动作的SGD方法。我们为AutoSGD及其确定性版本（用于标准梯度下降）提供了收敛性理论支持。实验证明，该方法在多种传统优化问题和机器学习任务中均表现出色。
原文摘要 · Abstract (English)
The learning rate is an important tuning parameter for stochastic gradient descent (SGD) and can greatly influence its performance. However, appropriate selection of a learning rate schedule across all iterations typically requires a non-trivial amount of user tuning effort. To address this, we introduce AutoSGD: an SGD method that automatically determines whether to increase or decrease the learning rate at a given iteration and then takes appropriate action. We introduce theory supporting the convergence of AutoSGD, along with its deterministic counterpart for standard gradient descent. Empirical results suggest strong performance of the method on a variety of traditional optimization problems and machine learning tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。