当Adam的动量参数相等时，优化更稳定，因梯度缩放不变性更强。
Why Adam Works Better with $β_1 = β_2$: The Missing Gradient Scale Invariance Principle
- 通过梯度缩放不变性理论，证明β₁=β₂是实现一阶不变性的必要条件。
- 实验显示在视觉与语言任务中，该设定使梯度重缩放后更新更平滑。
- 为未来优化器设计提供简洁可靠的原则，适合研究者和工程师参考。
Adam在过去十年中一直是大规模训练的核心，但一个简单的现象仍未得到解释：当动量参数满足β₁=β₂时，验证性能和训练过程的定性行为均有提升。尽管近期有研究观察到这一模式，但尚无理论说明其原因。本文提出并形式化了‘梯度缩放不变性’这一结构特性，证明当且仅当β₁=β₂时，Adam具备一阶梯度缩放不变性。这一视角使平衡状态的Adam与多个近期显式强化尺度鲁棒更新的优化器设计原则直接对齐。实验在视觉与语言任务、不同架构上均验证了该理论：梯度重缩放对更新的影响在β₁=β₂时显著更平滑。结果为长期未解的Adam行为问题提供了统一解释，并为未来优化器设计提供了一个简洁有效的指导原则。
原文摘要 · Abstract (English)
Adam has been at the core of large-scale training for almost a decade, yet a simple empirical fact remains unaccounted for: both validation scores and the qualitative behaviour of the training runs improve when the momentum parameters satisfy $β_{1}=β_{2}$. Some recent studies have reported this pattern, but there is still no explanation for why this choice helps. We show that this choice is closely tied to a structural property that we refer to as \textit{gradient scale invariance}. We formalize this notion and prove that Adam becomes gradient scale invariant of first order if and only if $β_{1}=β_{2}$. This perspective places the balanced regime of Adam in direct alignment with the design principles underlying several recent optimizers that explicitly enforce scale-robust updates. The theory is supported by experiments across vision and language tasks, and across different architectural families, in which rescaling the gradient has a markedly smoother effect on the update when $β_{1}=β_{2}$. Overall, our results offer a coherent explanation for an open question in the behavior of Adam and provide a simple principle that helps guide the design of future optimizers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。