改进Adam的二阶矩衰减机制，让训练后期更像动量SGD。
AdamNX: An Adam improvement algorithm based on a novel exponential decay mechanism for the second-order moment estimate
- 设计可变衰减率，逐步减弱对更新尺度的修正。
- 在图像分类等任务上表现优于原版Adam，尤其在收敛后期。
- 适合追求稳定收敛的深度学习训练场景。
本文研究了Adam算法中二阶矩估计的指数衰减机制。我们提出AdamNX及一种随时间变化的二阶矩衰减率，该机制逐步弱化对更新尺度的修正作用。在分析假设下，这一机制使训练后期的更新行为趋近于动量SGD。我们在图像分类、目标检测和语义分割任务上进行了实验，涵盖多种配置与对比方法。结果未建立多种子统计效应、平坦性或泛化能力的证据，仅限于报告任务范围内的性能表现。代码已开源：https://github.com/mengzhu0308/AdamNX。
原文摘要 · Abstract (English)
This paper studies the exponential decay mechanism of the second-moment estimate in Adam. We propose AdamNX and a time-varying second-moment decay rate that gradually weakens the correction applied to the update scale. Under the assumptions used in our analysis, this mechanism makes the updates approach momentum-SGD-like behavior during the training plateau phase. We report results on the image-classification, object-detection, and semantic-segmentation tasks, configurations, and comparators included in this paper. These results do not establish multi-seed statistical effects, flatness, or generalization beyond the reported tasks. Our code is open-sourced at https://github.com/mengzhu0308/AdamNX.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。