arXiv:2502.00753math.OCcs.LG2025-02被引 9

提出新型平滑性定义,让镜面下降法在非欧空间也高效收敛。

Mirror Descent Under Generalized Smoothness

  • 引入ℓ*-平滑性,用一般范数及其对偶衡量海森矩阵。
  • 理论证明镜面下降法收敛速度媲美经典光滑情况。
  • 适用于非凸、复合优化,对大模型训练有启发意义。

光滑性是实现一阶优化快速收敛的关键。然而,现代机器学习中的许多优化问题涉及非光滑目标函数。近期研究通过放宽光滑性假设,允许梯度的利普希茨常数随梯度范数增长,从而涵盖实践中广泛的优化目标。尽管如此,现有广义光滑性仍局限于欧几里得几何(ℓ₂-范数),且仅在欧氏空间中提供理论保证。本文通过引入新的ℓ* -光滑性概念,以一般范数及其对偶衡量海森矩阵的范数,建立了镜面下降类算法的收敛性,其收敛速率与经典光滑性下一致。值得注意的是,我们提出了广义自绑定性质,通过控制次优间隙来界定梯度,成为收敛分析的核心。除确定性优化外,我们还为随机镜面下降建立了精确收敛率,达到经典光滑性下的最优水平。该理论还可推广至非凸和复合优化,可能为镜面下降的实际应用(如大语言模型的预训练与后训练)提供新思路。

原文摘要 · Abstract (English)

Smoothness is crucial for attaining fast rates in first-order optimization. However, many optimization problems in modern machine learning involve non-smooth objectives. Recent studies relax the smoothness assumption by allowing the Lipschitz constant of the gradient to grow with respect to the gradient norm, which accommodates a broad range of objectives in practice. Despite this progress, existing generalizations of smoothness are restricted to Euclidean geometry with $\ell_2$-norm and only have theoretical guarantees for optimization in the Euclidean space. In this paper, we address this limitation by introducing a new $\ell*$-smoothness concept that measures the norm of Hessians in terms of a general norm and its dual, and establish convergence for mirror-descent-type algorithms, matching the rates under the classic smoothness. Notably, we propose a generalized self-bounding property that facilitates bounding the gradients via controlling suboptimality gaps, serving as a principal component for convergence analysis. Beyond deterministic optimization, we establish sharp convergence for stochastic mirror descent, matching state-of-the-art under classic smoothness. Our theory also extends to non-convex and composite optimization, which may shed light on practical usages of mirror descent, including pre-training and post-training of LLMs.

优化算法镜面下降非凸优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。