对比两种二阶优化方法，OGR在非凸问题上收敛更快、效果更好。
Comparing BFGS and OGR for Second-Order Optimization
- 用在线梯度回归法估算梯度与位置关系，避免求逆矩阵
- 在非凸测试函数上，损失下降更快且最终更低
- 适合处理非凸优化问题的研究者或工程应用
由于高维度和计算成本，估计海森矩阵（尤其在神经网络训练中）是一个挑战。本文对比了经典的BFGS方法所采用的Sherman-Morrison更新机制与一种新方法——在线梯度回归（OGR）。BFGS在凸性假设下维护一个正定的海森近似；而OGR通过指数移动平均对梯度与位置进行回归，实现在线估计二阶导数，无需海森逆运算。与BFGS不同，OGR可估计一般（不一定正定）的海森矩阵，因而能处理非凸结构。我们在标准测试函数上评估两者，结果表明，OGR在非凸设置下收敛速度更快、损失更低。
原文摘要 · Abstract (English)
Estimating the Hessian matrix, especially for neural network training, is a challenging problem due to high dimensionality and cost. In this work, we compare the classical Sherman-Morrison update used in the popular BFGS method (Broy-den-Fletcher-Goldfarb-Shanno), which maintains a positive definite Hessian approximation under a convexity assumption, with a novel approach called Online Gradient Regression (OGR). OGR performs regression of gradients against positions using an exponential moving average to estimate second derivatives online, without requiring Hessian inversion. Unlike BFGS, OGR allows estimation of a general (not necessarily positive definite) Hessian and can thus handle non-convex structures. We evaluate both methods across standard test functions and demonstrate that OGR achieves faster convergence and improved loss, particularly in non-convex settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。