针对非光滑优化问题，提出更优收敛速率的梯度剪裁与自适应方法。
Methods for Convex $(L_0,L_1)$-Smooth Optimization: Clipping, Acceleration, and Adaptivity
- 基于$(L_0,L_1)$-光滑假设，改进梯度剪裁与Polyak步长法的收敛性。
- 在强凸条件下实现无指数依赖的线性收敛，优于已有结果。
- 适用于非光滑机器学习优化，适合关注鲁棒性与加速技巧的研究者。
由于机器学习中优化问题的非光滑性，近年来广义光滑性假设受到广泛关注。其中最流行的类型之一是$(L_0,L_1)$-光滑性（Zhang et al., 2020）。本文聚焦于(强)凸$(L_0,L_1)$-光滑函数类，推导了若干现有方法的新收敛保证。特别地，我们改进了带（平滑）梯度剪裁的梯度下降法以及带Polyak步长的梯度下降法的收敛速率。与已有结果不同，我们的速率不依赖标准光滑性假设，且避免了初始距离到解的指数依赖。我们还将这些结果扩展到过参数化假设下的随机情形，提出一种新的加速方法用于凸$(L_0,L_1)$-光滑优化，并推导出自适应梯度下降法（Malitsky and Mishchenko, 2020）的新收敛速率。
原文摘要 · Abstract (English)
Due to the non-smoothness of optimization problems in Machine Learning, generalized smoothness assumptions have been gaining a lot of attention in recent years. One of the most popular assumptions of this type is $(L_0,L_1)$-smoothness (Zhang et al., 2020). In this paper, we focus on the class of (strongly) convex $(L_0,L_1)$-smooth functions and derive new convergence guarantees for several existing methods. In particular, we derive improved convergence rates for Gradient Descent with (Smoothed) Gradient Clipping and for Gradient Descent with Polyak Stepsizes. In contrast to the existing results, our rates do not rely on the standard smoothness assumption and do not suffer from the exponential dependency from the initial distance to the solution. We also extend these results to the stochastic case under the over-parameterization assumption, propose a new accelerated method for convex $(L_0,L_1)$-smooth optimization, and derive new convergence rates for Adaptive Gradient Descent (Malitsky and Mishchenko, 2020).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。