arXiv:2511.19716math.NAcs.LG2025-11被引 5

提出优化SGD预条件矩阵的三大设计准则,提升训练稳定性与收敛速度。

Design Criteria for SGD Preconditioners: Local Conditioning, Noise Floors, and Basin Stability

  • 基于预条件矩阵M构建几何空间,量化收敛速率与噪声下限。
  • 揭示预条件矩阵影响局部稳定性的机制,明确保证盆地稳定性概率。
  • 适用于科学机器学习场景,对模型物理一致性和数值稳定有指导意义。

随机梯度下降(SGD)在训练后期常因各向异性曲率和梯度噪声而变慢。本文研究由对称正定矩阵 $\mathbf{M}$ 定义的几何空间中的预条件SGD,推导出收敛速率与随机噪声下限均受 $\mathbf{M}$-相关量控制:速率由 $\mathbf{M}$-度量下的有效条件数决定,下限由该条件数与预条件后噪声水平的乘积决定。针对非凸目标函数,建立了依赖于预条件器的盆地稳定性保证:当平滑性与盆地大小以 $\mathbf{M}$-范数衡量时,迭代点保持在良好局部区域的概率具有显式下界。该视角在科学机器学习(SciML)中尤为重要,因小训练损失与物理保真度、数值稳定性和约束满足密切相关。该框架适用于对角/自适应及曲率感知预条件器,并导出简单设计原则:选择 $\mathbf{M}$ 以改善局部条件并抑制噪声。在二次诊断问题及三个SciML基准测试上的实验验证了预测的速率-噪声行为。

原文摘要 · Abstract (English)

Stochastic Gradient Descent (SGD) often slows in the late stage of training due to anisotropic curvature and gradient noise. We analyze preconditioned SGD in the geometry induced by a symmetric positive definite matrix $\mathbf{M}$, deriving bounds in which both the convergence rate and the stochastic noise floor are governed by $\mathbf{M}$-dependent quantities: the rate through an effective condition number in the $\mathbf{M}$-metric, and the floor through the product of that condition number and the preconditioned noise level. For nonconvex objectives, we establish a preconditioner-dependent basin-stability guarantee: when smoothness and basin size are measured in the $\mathbf{M}$-norm, the probability that the iterates remain in a well-behaved local region admits an explicit lower bound. This perspective is particularly relevant in Scientific Machine Learning (SciML), where achieving small training loss under stochastic updates is closely tied to physical fidelity, numerical stability, and constraint satisfaction. The framework applies to both diagonal/adaptive and curvature-aware preconditioners and yields a simple design principle: choose $\mathbf{M}$ to improve local conditioning while attenuating noise. Experiments on a quadratic diagnostic and three SciML benchmarks validate the predicted rate-floor behavior.

优化算法SGD科学机器学习预条件

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。