根据梯度衰减自适应调整学习率和批量大小，提升SGD收敛速度。
Adaptive Batch Size and Learning Rate Scheduler for Stochastic Gradient Descent Based on Minimization of Stochastic First-order Oracle Complexity
- 依据梯度范数衰减动态调节批量大小与学习率。
- 在实验中实现比现有调度器更快的收敛速度。
- 适合追求高效训练的深度学习研究者使用。
小批量随机梯度下降（SGD）的收敛行为对批量大小和学习率设置极为敏感。近期理论研究表明，存在一个临界批量大小，可使随机一阶预言机（SFO）复杂度最小化，即达到经验损失函数驻点所需的期望梯度评估次数最少。本文提出一种自适应调度策略，利用该临界批量大小的理论发现来加速SGD。批量大小和学习率根据训练过程中全梯度范数的观测衰减情况进行调整。基于此策略的自适应联合调度器在实验中表现出优于现有调度器的收敛速度。
原文摘要 · Abstract (English)
The convergence behavior of mini-batch stochastic gradient descent (SGD) is highly sensitive to the batch size and learning rate settings. Recent theoretical studies have identified the existence of a critical batch size that minimizes stochastic first-order oracle (SFO) complexity, defined as the expected number of gradient evaluations required to reach a stationary point of the empirical loss function in a deep neural network. An adaptive scheduling strategy is introduced to accelerate SGD that leverages theoretical findings on the critical batch size. The batch size and learning rate are adjusted on the basis of the observed decay in the full gradient norm during training. Experiments using an adaptive joint scheduler based on this strategy demonstrated improved convergence speed compared with that of existing schedulers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。