提出用递增批量训练QHM，无需衰减学习率即可实现稳定收敛。
Both Asymptotic and Non-Asymptotic Convergence of Quasi-Hyperbolic Momentum using Increasing Batch Size
- 通过递增批量大小实现非渐近与渐近收敛
- 有限批量增加即能提升神经网络训练效果
- 适合追求高效训练的深度学习实践者
动量方法最初在凸目标函数的确定性设置中表现出优于随机梯度下降（SGD）的性能。然而，尽管其广泛应用于深度神经网络这一典型的随机非凸优化场景，其有效性仍缺乏充分理论支持。准双曲动量（QHM）是泛化多种动量方法的算法，有助于理解动量类算法的整体特性。本文首次为使用递增批量大小的小批量QHM提供了渐近与非渐近收敛结果。我们证明，实现渐近收敛需学习率衰减或批量大小递增；由于学习率衰减会损害非渐近收敛，因此采用不衰减学习率但递增批量大小的策略更为有效。实验表明，即使批量大小仅有有限增长，也能显著提升神经网络训练表现。
原文摘要 · Abstract (English)
Momentum methods were originally introduced for their superiority to stochastic gradient descent (SGD) in deterministic settings with convex objective functions. However, despite their widespread application to deep neural networks -- a representative case of stochastic nonconvex optimization -- the theoretical justification for their effectiveness in such settings remains limited. Quasi-hyperbolic momentum (QHM) is an algorithm that generalizes various momentum methods and has been studied to better understand the class of momentum-based algorithms as a whole. In this paper, we provide both asymptotic and non-asymptotic convergence results for mini-batch QHM with an increasing batch size. We show that achieving asymptotic convergence requires either a decaying learning rate or an increasing batch size. Since a decaying learning rate adversely affects non-asymptotic convergence, we demonstrate that using mini-batch QHM with an increasing batch size -- without decaying the learning rate -- can be a more effective strategy. Our experiments show that even a finite increase in batch size can provide benefits for training neural networks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。