用动量稳定随机牛顿法，小批量也能全局收敛。
Improving Stochastic Cubic Newton with Momentum
- 引入特殊动量降低梯度和海森矩阵估计的方差。
- 单样本迭代下仍能收敛到二阶驻点，理论证明有效。
- 适合追求高效低资源优化的机器学习研究者。
我们研究用于解决一般非凸优化问题的随机二阶方法。提出使用一种特殊的动量来稳定牛顿法中的随机梯度与海森矩阵估计。理论上证明，该动量可显著降低估计方差，并使算法在任意噪声水平下均可收敛。结合立方正则化技术，我们在仅使用每轮一个随机数据样本的情况下，证明了该方法在一般非凸问题上的全局收敛速率，可达到二阶驻点。这与现有大多数随机二阶方法需大批次才能收敛形成鲜明对比，首次实现非凸情形下随机立方牛顿法在任意批量大小下的全局收敛。此外，我们还展示了在凸随机问题上，带动量的正则化牛顿法具有更优的收敛速度。
原文摘要 · Abstract (English)
We study stochastic second-order methods for solving general non-convex optimization problems. We propose using a special version of momentum to stabilize the stochastic gradient and Hessian estimates in Newton's method. We show that momentum provably improves the variance of stochastic estimates and allows the method to converge for any noise level. Using the cubic regularization technique, we prove a global convergence rate for our method on general non-convex problems to a second-order stationary point, even when using only a single stochastic data sample per iteration. This starkly contrasts with all existing stochastic second-order methods for non-convex problems, which typically require large batches. Therefore, we are the first to demonstrate global convergence for batches of arbitrary size in the non-convex case for the Stochastic Cubic Newton. Additionally, we show improved speed on convex stochastic problems for our regularized Newton methods with momentum.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。