增大批量和学习率可加速SGD训练，提升收敛效率。
Increasing Both Batch Size and Learning Rate Accelerates Stochastic Gradient Descent
- 提出四种动态调整批量与学习率的调度策略。
- 第3、4种策略使梯度范数更快下降，收敛更优。
- 适合追求高效训练的深度学习研究者参考。
小批量随机梯度下降（SGD）的性能高度依赖于批量大小和学习率的设置，以最小化深度神经网络训练中的经验损失。本文对四种调度器进行了理论分析：(i) 固定批量大小与衰减学习率；(ii) 增大批量大小与衰减学习率；(iii) 增大批量大小与增大学习率；(iv) 增大批量大小与预热衰减学习率。结果表明，仅使用调度器(i)无法保证最小化经验损失的全梯度范数期望，而使用调度器(ii)、(iii)或(iv)则可以。此外，调度器(iii)和(iv)能显著加速SGD收敛。数值实验进一步验证，相较于调度器(i)和(ii)，使用调度器(iii)或(iv)能更快地最小化经验损失的全梯度范数。
原文摘要 · Abstract (English)
The performance of mini-batch stochastic gradient descent (SGD) strongly depends on setting the batch size and learning rate to minimize the empirical loss in training the deep neural network. In this paper, we present theoretical analyses of mini-batch SGD with four schedulers: (i) constant batch size and decaying learning rate scheduler, (ii) increasing batch size and decaying learning rate scheduler, (iii) increasing batch size and increasing learning rate scheduler, and (iv) increasing batch size and warm-up decaying learning rate scheduler. We show that mini-batch SGD using scheduler (i) does not always minimize the expectation of the full gradient norm of the empirical loss, whereas it does using any of schedulers (ii), (iii), and (iv). Furthermore, schedulers (iii) and (iv) accelerate mini-batch SGD. The paper also provides numerical results of supporting analyses showing that using scheduler (iii) or (iv) minimizes the full gradient norm of the empirical loss faster than using scheduler (i) or (ii).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。