解析Adam在非平稳优化中的表现,揭示其适应与遗忘的权衡机制。
Adapt or Forget: Provable Tradeoffs Between Adam and SGD in Nonstationary Optimization

- 基于梯度预处理与动量跟踪,分解出初始化、目标漂移等四类误差源。
- 在噪声主导时自适应调整可降低误差,在漂移主导时反而加剧不稳定性。
- 给出参数β₁、β₂与ε的显式依赖关系,解释Adam为何在分布变化时忽稳忽崩。
本文对非平稳随机目标下的Adam算法进行理论分析,区分两种情形:在自适应强单调性下实现欧氏追踪,以及在一般L-光滑目标下获得高概率投影平稳性保证。在追踪情形中,推导出有限时间期望与高概率界,精确分解为初始化、目标漂移、由β₁控制的一阶动量追踪误差,以及由β₂控制的预条件扰动四项。刻画了恒定与阶梯衰减学习率下达到不可逾越追踪下界的燃烧期。还证明了在分布偏移下,Adam的平均投影平稳性间隙存在高概率界。两类分析共同揭示噪声-漂移权衡:噪声主导时,一阶动量平均与自适应预条件可改善高概率误差;漂移主导时,过时动量信息与预条件扰动会放大非平稳代价,使普通SGD能达到更小的追踪下界。显式的(β₁, β₂, ε)依赖界明确了自适应步长在何时有益或有害,为Adam的实证不稳定与分布偏移下的稳定化提供了理论依据。
原文摘要 · Abstract (English)
We provide a theoretical analysis of Adam under non-stationary stochastic objectives, separating two regimes: Euclidean tracking under adaptive strong monotonicity of the Adam-preconditioned mean-gradient operator, and high-probability projected stationarity guarantees under general $L$-smooth objectives. In the tracking regime, we derive finite-time expected and high-probability bounds that decompose sharply into four components: initialization, objective drift, a first-moment tracking error governed by $β_1$, and a preconditioner perturbation governed by $β_2$. We characterize the burn-in time to reach Adam's irreducible tracking floor under constant and step-decay schedules. We also prove a high-probability bound on the average projected stationarity gap for Adam under distribution shift. Across both analyses, our bounds reveal a noise--drift tradeoff: in noise-dominated regimes, first-moment averaging and adaptive preconditioning can improve the high-probability error, whereas in drift-dominated regimes, stale first-moment information and preconditioner perturbations can compound the cost of nonstationarity, allowing vanilla SGD to achieve a smaller tracking floor. Our explicit $(β_1,β_2,ε)$-dependent bounds delineate when adaptive step-sizing is beneficial versus harmful, and provide a theoretical mechanism for Adam's empirical instability and stabilization under distribution shift.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。