统一了带时变动量的随机优化算法，理论更通用且适用于零阶方法。
Convergence of Momentum-Based Optimization Algorithms with Time-Varying Parameters
- 提出带时间相关动量的统一优化框架，涵盖SHB和SNAG
- 在梯度有偏且方差无界条件下证明算法收敛
- 理论可支撑零阶优化，适合研究高维黑箱优化者
本文提出一种统一的随机优化算法，引入随时间变化的动量项，使当前梯度不仅依赖于当前真实梯度，还依赖于前一迭代的真实梯度。该框架包含随机重球法（SHB）和随机Nesterov加速梯度法（SNAG）作为特例。动量项可随迭代次数动态变化。对随机梯度的假设为文献中最宽松：允许有偏，且条件方差可随时间无界增长，这一特性对零阶方法至关重要——即仅通过两次函数值评估估计梯度。本文给出了该统一算法收敛的充分条件，这些条件是经典Robbins-Monro与Kiefer-Wolfowitz-Blum条件的自然推广。此外，我们分析了文献中另一类时变动量的SHB方法，发现其不具实用性。
原文摘要 · Abstract (English)
In this paper, we present a unified algorithm for stochastic optimization that makes use of a "momentum" term; in other words, the stochastic gradient depends not only on the current true gradient of the objective function, but also on the true gradient at the previous iteration. Our formulation includes the Stochastic Heavy Ball (SHB) and the Stochastic Nesterov Accelerated Gradient (SNAG) algorithms as special cases. In addition, in our formulation, the momentum term is allowed to vary as a function of time (i.e., the iteration counter). The assumptions on the stochastic gradient are the most general in the literature, in that it can be biased, and have a conditional variance that grows in an unbounded fashion as a function of time. This last feature is crucial in order to make the theory applicable to "zero-order" methods, where the gradient is estimated using just two function evaluations. We present a set of sufficient conditions for the convergence of the unified algorithm. These conditions are natural generalizations of the familiar Robbins-Monro and Kiefer-Wolfowitz-Blum conditions for standard stochastic gradient descent. We also analyze another method from the literature for the SHB algorithm with a time-varying momentum parameter, and show that it is impracticable.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。