增大批量大小能加速SGD动量法收敛，且更省计算资源。
Increasing Batch Size Improves Convergence of Stochastic Gradient Descent with Momentum
- 采用递增批量大小的SGD动量法可更好逼近最优解。
- 实验表明递增批量比固定批量更快收敛，且降低算力消耗。
- 适合深度学习训练中追求高效收敛的研究者和工程师。
随机梯度下降结合动量（SGDM）在理论与实践中均被广泛研究。理论分析表明学习率与动量系数影响收敛性；实践发现批量大小显著影响性能。本文聚焦于恒定学习率与动量系数下的小批量SGDM，用于训练深度神经网络。理论上证明：固定批量无法始终最小化经验损失的梯度范数期望，而递增批量则能确保最小化，即提升收敛性。数值实验支持该结论，显示递增批量的SGDM收敛至平稳点更快，同时减少计算成本。相关优化器的Python实现已公开于https://github.com/iiduka-researches/NSHB_increasing_batchsize_acml25/。
原文摘要 · Abstract (English)
Stochastic gradient descent with momentum (SGDM), in which a momentum term is added to SGD, has been well studied in both theory and practice. The theoretical studies show that the settings of the learning rate and momentum weight affect the convergence of SGDM. Meanwhile, the practical studies have shown that the batch-size setting strongly affects the performance of SGDM. In this paper, we focus on mini-batch SGDM with a constant learning rate and constant momentum weight, which is frequently used to train deep neural networks. We show theoretically that using a constant batch size does not always minimize the expectation of the full gradient norm of the empirical loss in training a deep neural network, whereas using an increasing batch size definitely minimizes it; that is, an increasing batch size improves the convergence of mini-batch SGDM. We also provide numerical results supporting our analyses, indicating specifically that mini-batch SGDM with an increasing batch size converges to stationary points faster than with a constant batch size, while also reducing computational cost. Python implementations of the optimizers used in the numerical experiments are available at https://github.com/iiduka-researches/NSHB_increasing_batchsize_acml25/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。