改进自适应优化器初始化，提升训练稳定性和模型性能
Revisiting the Initial Steps in Adaptive Gradient Descent Optimization
- 用非零值初始化二阶矩估计，替代传统归零做法
- 实验显示新方法让Adam收敛更稳，性能接近最新优化器
- 适合训练Transformer等敏感模型的科研与工程人员
自适应梯度优化方法（如Adam）在深度神经网络训练中广泛应用，因其收敛速度快。然而，这类方法常面临泛化性能不如随机梯度下降（SGD）及训练不稳定的问题，尤其在训练Transformer模型时更为明显。本文指出，标准的二阶矩估计初始值为零（v₀=0）是导致上述问题的重要因素。为此，提出简单有效的解决方案：采用数据驱动或随机方式对二阶矩估计进行非零初始化。实验证明，该方法不仅显著提升了收敛稳定性，还改善了优化器最终性能。采用新初始化策略后，Adam的性能可与近期提出的多种自适应优化器变体相媲美。代码已公开于https://github.com/Walleclipse/Adam_Initialization。
原文摘要 · Abstract (English)
Adaptive gradient optimization methods, such as Adam, are prevalent in training deep neural networks across diverse machine learning tasks due to their ability to achieve faster convergence. However, these methods often suffer from suboptimal generalization compared to stochastic gradient descent (SGD) and exhibit instability, particularly when training Transformer models. In this work, we show the standard initialization of the second-order moment estimation ($v_0 =0$) as a significant factor contributing to these limitations. We introduce simple yet effective solutions: initializing the second-order moment estimation with non-zero values, using either data-driven or random initialization strategies. Empirical evaluations demonstrate that our approach not only stabilizes convergence but also enhances the final performance of adaptive gradient optimizers. Furthermore, by adopting the proposed initialization strategies, Adam achieves performance comparable to many recently proposed variants of adaptive gradient optimization methods. Our code is available at https://github.com/Walleclipse/Adam_Initialization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。