Adam的梯度记忆会拖慢优化，即使噪声有界。
Second-Moment Memory in Coordinatewise Adam
- 用坐标级分析揭示了Adam的二阶动量记忆会抑制优化进度。
- 在平滑凸问题中，优化停滞点下界为 $O(M_2^{-1/2})$，$M_2$ 是记忆长度。
- 适合关注Adam收敛机制与参数调优的研究者阅读。
Adam 在分母中维护过去平方梯度的移动平均，但这种记忆带来的优化代价尚未明确。本文表明，即使在有限方差的随机梯度下，二阶动量记忆本身也会抑制向最优解的进展。针对一个简单的两点预言机，初始化暂态后，期望的正归一化更新量为 $O(M_2^{-1/2})$，其中 $M_2 = (1 - β_2)^{-1}$ 代表二阶动量记忆长度。在给定的记忆和步长缩放条件下，将该方向性边界转化为光滑凸问题上归一化间隙的平均平稳性下界，同样为同阶 $O(M_2^{-1/2})$。当梯度噪声具有有限方差时，过长的二阶动量记忆反而会减缓优化过程。
原文摘要 · Abstract (English)
Adam retains a moving average of past squared gradients in its denominator, but the optimization cost of this memory is not well understood. We show that second-moment memory can itself suppress progress toward the optimum even under finite-variance stochastic gradients. For a simple two-point oracle, the expected positive normalized update is $O(M_2^{-1/2})$ after an initialization transient, where $M_2=(1-β_2)^{-1}$ is the second-moment memory length. We convert this directional bound, under the stated memory and stepsize scaling, into an average-stationarity lower bound of the same order on a smooth convex problem with normalized gap, smoothness, and variance. Long second-moment memory can slow optimization even when the gradient noise has finite variance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。