通过捕捉局部曲率加速训练，减少33%迭代次数。
Accelerated Gradient Descent for Faster Convergence with Minimal Overhead
- 用有限差分显式计算局部曲率，提升优化效率。
- 相比基线方法，平均减少33%训练轮次，精度不变。
- 开销与Adam相当，适合深度学习训练加速。
本文提出CT-AGD（Curvature-Tuned Accelerated Gradient Descent），一种用于深度学习非凸优化的加速梯度下降方法。该方法通过有限差分商显式捕捉局部曲率，并设计启发式策略降低随机小批量训练带来的噪声与偏差。CT-AGD的存储与计算开销与Adam等自适应梯度方法相当。大量实验表明，其在保持与基线一阶方法相同精度的前提下，平均减少33%的训练轮次。
原文摘要 · Abstract (English)
In this paper, we present CT-AGD (Curvature-Tuned Accelerated Gradient Descent), an optimization method for non-convex optimization problems in deep learning training tasks. CT-AGD is a general boosting procedure that accelerates first-order methods by explicitly capturing the local curvature using finite-difference quotients, and the development of heuristics aimed at mitigating noise and bias introduced by stochastic mini-batch training. CT-AGD has a comparable storage and computational overhead as adaptive gradient methods such as Adam. Our extensive experiments demonstrate that CT-AGD achieves the same level of accuracy as the baseline first-order methods, yet reduces the required training epochs by 33% on average.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。