arXiv:2608.22129cs.LGcs.NA2026-08

提出可处理任意大张量的精确立方牛顿优化器,保持理论最优收敛速度。

Blockwise Stabilized Adaptive Cubic Regularization with Subsolvers via Recurrence

论文配图:Blockwise Stabilized Adaptive Cubic Regularization with Subsolvers via Recurrence
图 1 · 摘自论文原文
  • 每块独立求解真实海森矩阵上的立方模型,自适应调整正则化常数。
  • 在9140万参数隐式神经表示上,比Adam快且精度更高(133.5 dB vs 78.2 dB)。
  • 首次实现所有块上精确立方步,适合大规模高阶优化场景。

立方正则化牛顿法具有最优的 𝒪(ε⁻³/²) 全局收敛速率,但密集子问题求解限制了可行块大小。现有可扩展变体用对角、低秩、克罗内克分解或随机投影代理块曲率,通常放弃精确立方步。本文提出一种分块优化器,在每个参数张量上基于真实块海森矩阵最小化独立立方模型,采用每块自适应立方正则化常数和全损失单调性约束。通过兰佐斯构建的克雷洛夫子空间实现矩阵自由计算,证明该步确实最小化立方模型。理论保证 𝒪(ε⁻³/²) 迭代复杂度、二阶最优性及每块单调下降。评估了四种外层方案,对比原ARC优化器、其他近期立方牛顿方法、Adam、SOAP和L-BFGS。在9140万参数隐式神经表示(INR)上,本文方法是唯一在每块保持精确立方步的立方牛顿方法。在FINER 2D图像拟合任务中,所提的ARC-φ₁达到133.5 dB峰值信噪比,而调优后的Adam约70分钟后仅达78.2 dB,此时ARC-φ₁已达95.6 dB。

原文摘要 · Abstract (English)

Cubic regularized Newton methods have the optimal $\mathcal{O}(ε^{-3/2})$ global rate, but a dense subproblem solve limits the feasible block size. Scalable Cubic Newton variants replace the true block curvature with a diagonal, low-rank, Kronecker-factored, or sketched surrogate and, most often, give up the exact cubic step. We introduce a blockwise optimizer that minimizes an independent cubic model per parameter tensor over the true block Hessian, under a per-block adaptive cubic constant and a monotone guard on the full loss. Arbitrarily large tensors are handled matrix-free in a Lanczos-built Krylov subspace, where we prove that the step minimizes the cubic model. The theory also supplies the $\mathcal{O}(ε^{-3/2})$ iteration complexity bound, a second-order guarantee, and monotone per-block descent. Four variants of this outer scheme are evaluated against the original adaptive regularization with cubics (ARC) optimizer, some other recent cubic Newton variants, Adam, SOAP, and L-BFGS. On a 91.4M-parameter implicit neural representation (INR), the variants introduced in this work are the only evaluated here cubic Newton methods whose steps stay exact on every block. Run to full convergence on FINER 2D image fitting, one of the ARC variants introduced here, ARC-$φ_1$, reaches 133.5 dB peak signal-to-noise ratio, while tuned Adam plateaus at 78.2 dB after about 70 minutes. In that time ARC-$φ_1$ reaches 95.6 dB.

优化算法牛顿法深度学习高阶优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。