用随机步长修正动量法偏差，提升优化收敛速度。
RanSOM: Second-Order Momentum with Randomized Scaling for Constrained and Unconstrained Optimization
- 用随机步长替代固定步长，通过单次海森向量积无偏估计动量偏差。
- 理论证明可达到最优 $\mathcal{O}(ε^{-3})$ 收敛率，适用于重尾噪声场景。
- 适用于无约束（RanSOM-E）和有约束（RanSOM-B）优化，保持可行性。
动量方法（如Polyak的Heavy Ball）是训练深度网络的标准方法，但在随机设置下受曲率偏差影响，收敛速率受限于次优的 $\mathcal{O}(ε^{-4})$。现有修正方法通常需要昂贵的辅助采样或严格的光滑性假设。本文提出统一框架 RanSOM，通过将确定性步长替换为均值为 $η_t$ 的随机步长，利用Stein型恒等式，仅用一次与梯度联合计算的海森向量积即可获得动量偏差的精确无偏估计，避免额外查询。我们实例化两个算法：RanSOM-E（无约束，指数分布步长）和 RanSOM-B（有约束，贝塔分布步长以严格保持可行性）。理论分析表明，RanSOM 在标准有界噪声下恢复最优 $\mathcal{O}(ε^{-3})$ 收敛率，并在重尾噪声情形（$p \in (1, 2]$）也达到最优速率。
原文摘要 · Abstract (English)
Momentum methods, such as Polyak's Heavy Ball, are the standard for training deep networks but suffer from curvature-induced bias in stochastic settings, limiting convergence to suboptimal $\mathcal{O}(ε^{-4})$ rates. Existing corrections typically require expensive auxiliary sampling or restrictive smoothness assumptions. We propose \textbf{RanSOM}, a unified framework that eliminates this bias by replacing deterministic step sizes with randomized steps drawn from distributions with mean $η_t$. This modification allows us to leverage Stein-type identities to compute an exact, unbiased estimate of the momentum bias using a single Hessian-vector product computed jointly with the gradient, avoiding auxiliary queries. We instantiate this framework in two algorithms: \textbf{RanSOM-E} for unconstrained optimization (using exponentially distributed steps) and \textbf{RanSOM-B} for constrained optimization (using beta-distributed steps to strictly preserve feasibility). Theoretical analysis confirms that RanSOM recovers the optimal $\mathcal{O}(ε^{-3})$ convergence rate under standard bounded noise, and achieves optimal rates for heavy-tailed noise settings ($p \in (1, 2]$).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。