提出一种更稳定的梯度下降方法，能避免传统加速方法在复杂问题中发散。
Robust Gradient Descent via Heavy-Ball Momentum with Predictive Extrapolation
- 用局部泰勒近似预测未来梯度方向，替代累积历史梯度
- 在条件数50的病态问题上119步收敛，传统方法发散
- 适合非凸问题，对超参数不敏感，内存开销仅O(d)
Nesterov加速梯度（NAG）在条件良好问题上收敛快，但在病态或非凸问题上因动量累积过强而发散。本文提出重球合成梯度外推法（HB-SGE），将重球动量与预测梯度外推结合。不同于传统方法累积历史梯度，HB-SGE通过局部泰勒近似估计未来梯度方向，实现自适应加速并保持稳定。理论证明其对强凸函数有收敛保证，实验证明在条件数κ=50的病态二次问题上，HB-SGE于119次迭代收敛，而SGD和NAG均发散；在非凸Rosenbrock函数上，HB-SGE于2718次迭代收敛，传统动量方法在10步内即发散。虽在条件良好问题上不及NAG快，但HB-SGE在多种场景下显著优于SGD，仅需O(d)内存开销，且超参数与标准动量一致。
原文摘要 · Abstract (English)
Accelerated gradient methods like Nesterov's Accelerated Gradient (NAG) achieve faster convergence on well-conditioned problems but often diverge on ill-conditioned or non-convex landscapes due to aggressive momentum accumulation. We propose Heavy-Ball Synthetic Gradient Extrapolation (HB-SGE), a robust first-order method that combines heavy-ball momentum with predictive gradient extrapolation. Unlike classical momentum methods that accumulate historical gradients, HB-SGE estimates future gradient directions using local Taylor approximations, providing adaptive acceleration while maintaining stability. We prove convergence guarantees for strongly convex functions and demonstrate empirically that HB-SGE prevents divergence on problems where NAG and standard momentum fail. On ill-conditioned quadratics (condition number $κ=50$), HB-SGE converges in 119 iterations while both SGD and NAG diverge. On the non-convex Rosenbrock function, HB-SGE achieves convergence in 2,718 iterations where classical momentum methods diverge within 10 steps. While NAG remains faster on well-conditioned problems, HB-SGE provides a robust alternative with speedup over SGD across diverse landscapes, requiring only $O(d)$ memory overhead and the same hyperparameters as standard momentum.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。