无需调参的自适应梯度方法，提升随机优化收敛性。
Stochastic Adaptive Gradient Descent Without Descent
- 基于一阶随机梯度信息，动态调整步长以捕捉局部几何。
- 理论证明在多种假设下可保证收敛，实测性能媲美调优基线。
- 适合对超参敏感或追求自动化优化的场景。
我们提出一种新的自适应步长策略，用于带随机梯度的凸优化问题。该方法仅利用一阶随机梯度信息，无需任何超参数调优即可捕捉目标函数的局部几何结构。该方法源自对无下降自适应梯度下降法在随机设置下的理论化改进。我们在多种假设下证明了采用该步长的随机梯度下降的收敛性，并在实验中展示其性能可与经过调优的基线方法相媲美。
原文摘要 · Abstract (English)
We introduce a new adaptive step-size strategy for convex optimization with stochastic gradient that exploits the local geometry of the objective function only by means of a first-order stochastic oracle and without any hyper-parameter tuning. The method comes from a theoretically-grounded adaptation of the Adaptive Gradient Descent Without Descent method to the stochastic setting. We prove the convergence of stochastic gradient descent with our step-size under various assumptions, and we show that it empirically competes against tuned baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。