新优化算法F-CMA提升大规模训练速度与精度，兼顾高效与理论保证。
Beyond adaptive gradient: Fast-Controlled Minibatch Algorithm for large-scale optimization
- 基于随机重排与线搜索，每轮迭代确保损失下降。
- 训练时间减少最多68%，单轮效率提升20%，准确率最高提高5%。
- 适合追求高效训练的深度学习研究者和工程实践者。
自适应梯度方法因收敛快、对超参数不敏感而被广泛采用，但存在内存开销大、收敛理论不清晰等问题。为此，本文提出F-CMA——一种基于随机重排与线搜索的快速控制小批量算法，具备充分下降条件和确定性全局收敛至驻点的证明。在卷积神经网络与视觉变压器模型的分类任务中集成测试，结果表明：训练时间最多缩短68%，每轮效率提升20%，模型准确率最高增加5%，显著优于主流优化器。
原文摘要 · Abstract (English)
Adaptive gradient methods have been increasingly adopted by deep learning community due to their fast convergence and reduced sensitivity to hyper-parameters. However, these methods come with limitations, such as increased memory requirements for elements like moving averages and a poorly understood convergence theory. To overcome these challenges, we introduce F-CMA, a Fast-Controlled Mini-batch Algorithm with a random reshuffling method featuring a sufficient decrease condition and a line-search procedure to ensure loss reduction per epoch, along with its deterministic proof of global convergence to a stationary point. To evaluate the F-CMA, we integrate it into conventional training protocols for classification tasks involving both convolutional neural networks and vision transformer models, allowing for a direct comparison with popular optimizers. Computational tests show significant improvements, including a decrease in the overall training time by up to 68%, an increase in per-epoch efficiency by up to 20%, and in model accuracy by up to 5%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。