提出简单且有理论保证的自适应优化算法，无需调学习率
Towards Simple and Provable Parameter-Free Adaptive Gradient Methods
- 设计简洁的参数无关版AdaGrad和Adam，自动调节学习率
- 理论证明新算法在凸优化中收敛速度与原版相当
- 适合追求稳定训练且不想调参的研究者和工程师
AdaGrad和Adam等自适应优化算法通过动态调整学习率显著提升了深度模型的训练效果。然而，实践中仍需手动调参，导致效率低下。为解决此问题，近期研究致力于开发无需调参的“参数无关”算法。但现有版本往往过于复杂或缺乏严格的收敛性保证。本文提出AdaGrad++和Adam++，两个新颖且简洁的参数无关变体，并证明其在凸优化中可达到与原始算法相当的收敛速率，且不依赖预设学习率。实验结果表明，Adam++在多种深度学习任务中表现优异，具备竞争力。
原文摘要 · Abstract (English)
Optimization algorithms such as AdaGrad and Adam have significantly advanced the training of deep models by dynamically adjusting the learning rate during the optimization process. However, ad-hoc tuning of learning rates poses a challenge and leads to inefficiencies in practice. To address this issue, recent research has focused on developing ``parameter-free'' algorithms that operate effectively without the need for learning rate tuning. Despite these efforts, existing parameter-free variants of AdaGrad and Adam tend to be overly complex and/or lack formal convergence guarantees. In this paper, we present AdaGrad++ and Adam++, novel and simple parameter-free variants of AdaGrad and Adam with convergence guarantees. We prove that AdaGrad++ achieves comparable convergence rates to AdaGrad in convex optimization without predefined learning rate assumptions. Similarly, Adam++ matches the convergence rate of Adam without relying on any conditions on the learning rates. Experimental results across various deep learning tasks validate the competitive performance of Adam++.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。