通过新方法证明:动态调整学习率和批量大小能显著加速SGDM收敛。
Accelerating SGDM via Learning Rate and Batch Size Schedules: A Lyapunov-Based Analysis
- 提出简单新李雅普诺夫函数分析SGDM在动态超参数下的收敛性
- 发现批量大小递增可保证收敛,且同步增大批量与学习率收敛更快
- 实验验证动态调度比固定超参数快,预热策略表现最优
我们通过引入一种新颖且更简单的李雅普诺夫函数,分析了随机梯度下降带动量(SGDM)在动态学习率和批量大小调度下的收敛行为。将现有理论框架扩展至深度学习中常用的三种实际调度策略:固定批量大小配合衰减学习率、递增批量大小配合衰减学习率,以及同时递增批量大小和学习率。结果揭示出明确的收敛层级关系:固定批量大小无法在该李雅普诺夫分析下保证期望梯度范数收敛,而递增批量大小可以实现收敛,同时递增批量大小和学习率则能实现可证明更快的衰减速率。实验验证了理论结论,表明动态调度的SGDM在收敛速度上显著优于固定超参数版本。此外,对预热调度的评估也显示其在收敛行为上优于其他策略。
原文摘要 · Abstract (English)
We analyze the convergence behavior of stochastic gradient descent with momentum (SGDM) under dynamic learning-rate and batch-size schedules by introducing a novel and simpler Lyapunov function. We extend the existing theoretical framework to cover three practical scheduling strategies commonly used in deep learning: a constant batch size with a decaying learning rate, an increasing batch size with a decaying learning rate, and an increasing batch size with an increasing learning rate. Our results reveal a clear hierarchy in convergence: a constant batch size does not guarantee convergence of the expected gradient norm under our Lyapunov-based analysis, whereas an increasing batch size does, and simultaneously increasing both the batch size and learning rate achieves a provably faster decay. Empirical results validate our theory, showing that dynamically scheduled SGDM significantly outperforms its fixed-hyperparameter counterpart in convergence speed. We also evaluated a warmup schedule in experiments, which empirically outperformed all other strategies in convergence behavior.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。