提出新优化器C-Adam，理论证明收敛性并实证有效。
A Theoretical and Experimental Study of a Novel Adaptive Learning Algorithm
- 基于视线法设计新型自适应优化器C-Adam
- 理论证明该方法可保证收敛，实验验证性能优于传统方法
- 适合追求收敛性保障的深度学习优化研究者
机器学习算法的核心在于以较低计算成本和较少震荡最小化损失函数。尽管基于自适应学习率的优化器在实际任务中广泛应用，但它们并不保证收敛性，因此后续提出了AMSGrad来研究Adam的非收敛行为。本文对Adam、AMSGrad等主流自适应优化方法进行了批判性回顾，重点分析其基本设计思想。为克服上述优化器的局限性，本文提出一种基于视线法的新优化器变体C-Adam。同时提供了收敛性的理论证明，并通过多个真实场景的数值实验验证了其有效性。
原文摘要 · Abstract (English)
A crucial component of machine learning algorithms is minimizing loss functions with less computational cost and less oscillations. While adaptive learning rate-based optimizers have been widely used for real-world tasks, they do not guarantee convergence, which is why AMSGrad was later introduced to investigate the non-convergence behaviour of Adam. In this paper, popular adaptive optimization methods like Adam and AMSGrad are critically reviewed with an emphasis on their fundamental design concepts. To address limitations of the above mentioned optimizers, a new optimizer variant, C-Adam, is proposed based on the line of sight approach. A theoretical proof for convergence is also provided and the optimizer is validated through a number of real-life based numerical experiments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。