让Adam优化器的动量参数自动匹配训练记忆时长,提升鲁棒性。
Refresh-Scaling the Memory of Balanced Adam
- 将动量参数β转化为记忆时长概念,通过有效学习周期计算刷新次数
- 设定刷新次数约1000时,11个实验中所有结果均在最优值1%内
- 相比固定β=0.944,最差情况损失差距减少33.4%,适合大规模训练
近期研究表明,当Adam的动量参数满足β₁=β₂时,其性能更稳定,仅剩一个可调参数。我们提出,该参数β不应视为无量纲常数,而应理解为统计记忆时长H_β=(1-β)⁻¹。基于验证轨迹估计的有效学习时长T_ES,我们定义刷新次数R_β=(1-β)T_ES,衡量训练有效阶段内优化器重置内部统计的次数。在11个视觉与语言实验中,设置R_β≈1000可自适应选择不同β值,显著优于最佳固定β基线。相比最强固定β=0.944,该刷新规则使最差情况验证损失相对差距降低33.4%,且全部11次运行均在验证最优值1%以内。结果表明,平衡Adam的剩余超参更宜视为记忆尺度变量,而非固定常数,为优化器缩放提供了预算感知视角,并将动量纳入学习动态而非静态默认。
原文摘要 · Abstract (English)
Recent evidence suggests that Adam performs robustly when its momentum parameters are tied, $β_1=β_2$, reducing the optimizer to a single remaining parameter. However, how this parameter should be set remains poorly understood. We argue that, in balanced Adam, $β$ should not be treated as a dimensionless constant: it defines a statistical memory horizon $H_β=(1-β)^{-1}$. In terms of the effective learning horizon $T_{\mathrm{ES}}$, estimated from the validation trajectory, we study the refresh count $R_β=(1-β)T_{\mathrm{ES}}$, which measures how many times Adam renews its internal statistics during the useful phase of training. Across 11 vision and language experiments, we find that choosing $β$ so that $R_β\approx1000$ selects different $β$ values depending on the training scale, yet improves robustness over the best fixed-beta baseline. Compared with the strongest fixed choice $β=0.944$, the refresh rule improves worst-case robustness, reducing the maximum relative gap in validation loss by 33.4\%, while bringing all 11 runs within 1\% of their validation oracle. These results suggest that the remaining hyperparameter of balanced Adam is more naturally viewed as a memory-scale variable than as a fixed constant. This provides a simple budget-aware perspective on optimizer scaling and opens a path toward treating Adam's momentum as part of the learning dynamics rather than as a static default.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。