提出最优学习率与批大小增长策略，降低深度学习训练复杂度。
Optimal Growth Schedules for Batch Size and Learning Rate in SGD that Reduce SFO Complexity
- 基于SFO复杂度理论推导出批大小和学习率的最优增长规律
- 实验验证该策略可显著减少达到收敛所需的梯度计算量
- 适合大规模深度学习模型的高效训练场景
深度学习模型的迅猛发展带来了巨大性能提升，但也引发严重的计算瓶颈。其中，随机梯度方法中的批大小与学习率调度是影响训练效率的关键因素。然而，不当的调度会降低优化效率并损害泛化能力。受近期理论发现启发，本文研究了训练过程中批大小与学习率应如何增长，以平衡效率与收敛性。基于随机一阶预言机（SFO）复杂度——即达到经验损失ε-近似驻点所需期望梯度评估次数——进行分析，理论上推导出能降低SFO复杂度的批大小与学习率最优增长策略，并通过大量实验加以验证。研究成果为大规模深度学习的可扩展、高效训练提供了理论依据与实践指导。
原文摘要 · Abstract (English)
The unprecedented growth of deep learning models has enabled remarkable advances but introduced substantial computational bottlenecks. A key factor contributing to training efficiency is batch-size and learning-rate scheduling in stochastic gradient methods. However, naive scheduling of these hyperparameters can degrade optimization efficiency and compromise generalization. Motivated by recent theoretical insights, we investigated how the batch size and learning rate should be increased during training to balance efficiency and convergence. We analyzed this problem on the basis of stochastic first-order oracle (SFO) complexity, defined as the expected number of gradient evaluations needed to reach an $ε$-approximate stationary point of the empirical loss. We theoretically derived optimal growth schedules for the batch size and learning rate that reduce SFO complexity and validated them through extensive experiments. Our results offer both theoretical insights and practical guidelines for scalable and efficient large-batch training in deep learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。