提出一种高效在线牛顿法,实现与一阶方法相当的计算复杂度。
Inference of Online Newton Methods with Nesterov's Accelerated Sketching

- 用带加速的随机投影法近似求解牛顿方向,降低复杂度至O(d²)
- 证明了迭代序列几乎必然收敛,且末次迭代渐近正态
- 适用于需要实时不确定性量化的大数据在线学习场景
流式数据下的可靠决策依赖于在线方法的不确定性量化。尽管一阶方法可高效更新迭代点,其推断仍需维护协方差矩阵,带来O(d²)的时间与内存开销,且对病态问题和噪声异质性敏感。这为更鲁棒的二阶方法提供了机会,但其瓶颈在于求解牛顿系统需O(d³)复杂度。本文研究一种带有海森平均的在线牛顿法,每步通过带内斯特罗夫加速的随机投影求解器近似计算牛顿方向,使复杂度降至O(d²),与一阶方法相当。针对该方法,我们量化了由随机数据与随机计算共同引起的不确定性。在标准光滑性和矩条件下,建立了全局几乎必然收敛性,证明末次迭代渐近正态,其极限协方差由一个李雅普诺夫方程刻画,并设计了具有非渐近收敛保证的全在线协方差估计器。还揭示了该不确定性量化与无加速的精确及随机牛顿方法之间的联系。回归模型上的大量实验验证了该方法在在线推断中的优越性。
原文摘要 · Abstract (English)
Reliable decision-making with streaming data requires principled uncertainty quantification of online methods. While first-order methods enable efficient iterate updates, their inference procedures still require updating proper (covariance) matrices, incurring $O(d^2)$ time and memory complexity, and are sensitive to ill-conditioning and noise heterogeneity of the problem. This costly inference task offers an opportunity for more robust second-order methods, which are, however, bottlenecked by solving Newton systems with $O(d^3)$ complexity. In this paper, we address this gap by studying an online Newton method with Hessian averaging, where the Newton direction at each step is approximately computed using a sketch-and-project solver with Nesterov's acceleration, matching $O(d^2)$ complexity of first-order methods. For the proposed method, we quantify its uncertainty arising from both random data and randomized computation. Under standard smoothness and moment conditions, we establish global almost-sure convergence, prove asymptotic normality of the last iterate with a limiting covariance characterized by a Lyapunov equation, and develop a fully online covariance estimator with non-asymptotic convergence guarantees. We also connect the resulting uncertainty quantification to that of exact and sketched Newton methods without Nesterov's acceleration. Extensive experiments on regression models demonstrate the superiority of the proposed method for online inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。