提出新型自适应优化器OptEMA,无需调参即可在无噪声时自动达到最优收敛速度。
OptEMA: Adaptive Exponential Moving Average for Stochastic Optimization with Zero-Noise Optimality
- 采用动态调整的一阶动量衰减机制,实现闭环自适应优化
- 在零噪声下收敛率逼近最优的T^{-1/2},无需手动调参
- 适用于对收敛性要求高且缺乏先验信息的深度学习场景
指数移动平均(EMAs)是Adam等自适应优化器的核心组件。然而,现有对Adam类方法的分析在零噪声情形下常得次优保证,依赖开环参数调度或需预先知晓光滑性常数。为此,本文提出OptEMA及其两种互补变体:OptEMA-M采用自适应递减的一阶动量衰减、固定二阶动量衰减;OptEMA-V则角色互换。其核心为修正的AdaGrad-Norm系数调度,使算法闭环且无需利普希茨常数参数化。在函数下界、无偏性、有界方差、平均光滑性及控制自适应归一化的有界随机梯度条件下,证明两种变体均达到统一噪声自适应率 ilde{/mathcal{O}}(T^{-1/2} + σ^{1/2}T^{-1/4})。在零噪声情形下,该界自动简化为近乎最优的确定性率 ilde{/mathcal{O}}(T^{-1/2}),无需人工调参。
原文摘要 · Abstract (English)
Exponential moving averages (EMAs) are a central component of widely used adaptive optimizers such as Adam. However, existing analyses of Adam-style methods often yield suboptimal guarantees in the zero-noise regime, rely on open-loop parameter schedules, or require prior knowledge of smoothness constants. Motivated by these limitations, we introduce OptEMA and analyze two complementary variants: OptEMA-M, which applies an adaptive, decreasing EMA coefficient to the first moment with a fixed second-moment decay, and OptEMA-V, which swaps these roles. At the heart of these variants is a Corrected AdaGrad-Norm coefficient schedule. This formulation renders OptEMA algorithmically closed-loop and Lipschitz-free, meaning its effective stepsizes are trajectory-dependent and require no parameterization via the Lipschitz constant. Under lower-boundedness, unbiasedness, bounded variance, average smoothness, and a bounded stochastic-gradient condition used to control the adaptive normalizers, we prove that both variants achieve the unified noise-adaptive rate $\tilde{\mathcal{O}} \left(T^{-1/2}+σ^{1/2}T^{-1/4}\right)$ for the averaged gradient norm. In the zero-noise regime, these bounds automatically reduce to the nearly optimal deterministic rate $\widetilde{\mathcal{O}}(T^{-1/2})$ without manual hyperparameter retuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。