通过预判未来梯度位置，提升动量优化器收敛速度
Overshoot: Taking advantage of future gradients in momentum-based stochastic optimization
- 在动量方向上预估未来权重点计算梯度，而非当前点
- 相比标准与Nesterov动量，平均减少15%以上优化步数
- 兼容主流优化器，内存和计算开销极小，适合实际部署
Overshoot是一种新型的动量式随机梯度下降优化方法，旨在超越标准动量与Nesterov动量的表现。传统动量方法在更新模型前，将历史梯度与当前模型权重处的梯度合并。Overshoot则改在动量方向上偏移后的权重点计算梯度，虽牺牲了当前权重的即时梯度信息，但更贴近未来更新时的有用梯度。实验表明，将该思想融入SGD with momentum和Adam等优化器后，平均可节省至少15%的优化步数，且在多种任务上持续优于标准与Nesterov动量。该方法可无缝集成至主流动量优化器，无需额外内存，计算开销极小。
原文摘要 · Abstract (English)
Overshoot is a novel, momentum-based stochastic gradient descent optimization method designed to enhance performance beyond standard and Nesterov's momentum. In conventional momentum methods, gradients from previous steps are aggregated with the gradient at current model weights before taking a step and updating the model. Rather than calculating gradient at the current model weights, Overshoot calculates the gradient at model weights shifted in the direction of the current momentum. This sacrifices the immediate benefit of using the gradient w.r.t. the exact model weights now, in favor of evaluating at a point, which will likely be more relevant for future updates. We show that incorporating this principle into momentum-based optimizers (SGD with momentum and Adam) results in faster convergence (saving on average at least 15% of steps). Overshoot consistently outperforms both standard and Nesterov's momentum across a wide range of tasks and integrates into popular momentum-based optimizers with zero memory and small computational overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。