arXiv:2511.17660cs.LGcs.AI2025-11

用低精度计算让牛顿法更高效,训练更快更准

Frugality in second-order optimization: floating-point approximations for Newton's method

  • 用混合精度计算优化牛顿法,降低算力消耗
  • 在澳大利亚和MUSH数据集上优于Adam,收敛更快
  • 只需部分计算二阶导数,适合资源有限场景

机器学习训练中最小化损失函数至关重要。尽管一阶方法应用广泛,但牛顿法等高阶方法可实现更高精度和更快收敛,常因计算成本被忽略。本文分析有限精度算术对牛顿步的影响,建立混合精度牛顿优化器的收敛定理,包含‘准’与‘不精确’变体,提供收敛保证及解精度的先验估计。在标准回归基准测试中,所提方法在澳大利亚和MUSH数据集上表现优于Adam。第二部分提出GN_k——一种广义高斯-牛顿法,支持二阶导数的部分计算,在回归任务中性能接近完整牛顿法,却大幅减少导数评估次数。

原文摘要 · Abstract (English)

Minimizing loss functions is central to machine-learning training. Although first-order methods dominate practical applications, higher-order techniques such as Newton's method can deliver greater accuracy and faster convergence, yet are often avoided due to their computational cost. This work analyzes the impact of finite-precision arithmetic on Newton steps and establishes a convergence theorem for mixed-precision Newton optimizers, including "quasi" and "inexact" variants. The theorem provides not only convergence guarantees but also a priori estimates of the achievable solution accuracy. Empirical evaluations on standard regression benchmarks demonstrate that the proposed methods outperform Adam on the Australian and MUSH datasets. The second part of the manuscript introduces GN_k, a generalized Gauss-Newton method that enables partial computation of second-order derivatives. GN_k attains performance comparable to full Newton's method on regression tasks while requiring significantly fewer derivative evaluations.

牛顿法二阶优化混合精度高效训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。