EXAdam改进Adam优化器,提升收敛速度与精度。
EXAdam: The Power of Adaptive Cross-Moments
- 引入新去偏项和梯度加速机制,增强对损失曲面的响应能力。
- 在CIFAR-10上收敛快38.46%,准确率提升1.96%~2.17%。
- 适合追求高效训练的深度学习研究者与工程实践者。
本文提出EXAdam(扩展Adam),一种基于广泛使用的Adam优化器的新算法。EXAdam引入两项关键改进:(1) 新的去偏项以提升矩估计精度;(2) 基于梯度的加速机制,增强对当前损失景观的响应性。这两项创新协同作用,解决了原始Adam的局限性,可能带来更好的收敛性、更强的鞍点逃逸能力以及对超参数选择更高的鲁棒性,尽管后者仍需进一步验证。我们对EXAdam各组件及其相互作用进行了理论分析,突显其在复杂优化空间中的潜在优势。实验表明,相较于Adam,EXAdam在卷积神经网络于CIFAR-10上的训练中实现38.46%的更快收敛,并在训练、验证和测试准确率上分别提升1.96%、2.17%和1.17%。尽管结果有前景,但需在多样任务上进一步验证其有效性。总体而言,EXAdam代表了自适应优化技术的重要进展,为机器学习与人工智能领域的广泛应用带来潜力。
原文摘要 · Abstract (English)
This paper introduces EXAdam ($\textbf{EX}$tended $\textbf{Adam}$), a novel optimization algorithm that builds upon the widely-used Adam optimizer. EXAdam incorporates two key enhancements: (1) new debiasing terms for improved moment estimation and (2) a gradient-based acceleration mechanism for increased responsiveness to the current loss landscape. These innovations work synergistically to address limitations of the original Adam algorithm, potentially offering improved convergence properties, enhanced ability to escape saddle points, and potentially greater robustness to hyperparameter choices, though this requires further investigation. We provide a theoretical analysis of EXAdam's components and their interactions, highlighting the algorithm's potential advantages in navigating complex optimization landscapes. Empirical evaluations demonstrate EXAdam's superiority over Adam, achieving 38.46% faster convergence and yielding improvements of 1.96%, 2.17%, and 1.17% in training, validation, and testing accuracies, respectively, when applied to a CNN trained on the CIFAR-10 dataset. While these results are promising, further empirical validation across diverse tasks is essential to fully gauge EXAdam's efficacy. Nevertheless, EXAdam represents a significant advancement in adaptive optimization techniques, with promising implications for a wide range of machine learning applications. This work aims to contribute to the ongoing development of more efficient, adaptive, and universally applicable optimization methods in the field of machine learning and artificial intelligence.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。