基于单侧霍尔德曲率自适应调整步长,提升非凸优化效率。
Learning from the Descent Direction: Adaptive Gradient Descent under One-Sided Hölder Regularity
- 仅约束下降不等式中的方向项,允许更灵活的步长控制。
- 在两类基准测试中均达到最低目标值与梯度范数,性能最优。
- 适合追求高效非凸优化的科研与工程人员使用。
研究在单侧霍尔德正则性下,对连续可微、可能非凸的目标函数进行自适应梯度下降。与经典霍尔德或利普希茨梯度假设不同,该条件仅限制下降不等式中出现的方向项,当梯度大幅变化方向与更新方向正交或有利时,可采用更保守的步长。提出一种基于正单侧霍尔德曲率估计的自适应标量步长方法,并结合简单充分下降保障机制。对于包含已接受更新段的凸区域上的非凸目标函数,证明了显式的最优迭代点平稳性界,其收敛速率由霍尔德指数决定。相较于预设递减步长方案,该方法能自适应局部下降几何。在两个全批量基准上评估,分别针对二分类问题与非凸霍尔德回归问题。在二分类任务中,该方法获得最低交叉熵、目标值与梯度范数,同时分类边界最大;在非凸霍尔德回归中,最终目标差距与梯度范数最低。结果表明,当全梯度变化被非阻碍下降方向放大时,单侧霍尔德曲率是有效的自适应步长信号。
原文摘要 · Abstract (English)
We study adaptive gradient descent for continuously differentiable, possibly nonconvex objectives under one-sided Hölder regularity. Unlike classical Hölder- or Lipschitz-gradient assumptions, which control the full gradient variation, our condition bounds only the directional term appearing in the descent inequality. This can allow less conservative step sizes when large gradient changes are orthogonal to, or favorable along, the update direction. We propose an adaptive scalar-step method based on an estimate of positive one-sided Hölder curvature, combined with a simple sufficient-decrease safeguard. For nonconvex objectives on a convex region containing the accepted update segments, we prove an explicit best-iterate stationarity bound with a rate determined by the Hölder exponent. Unlike predetermined diminishing step-size schemes, the method adapts to the local descent geometry. We evaluate the approach on two full-batch benchmarks designed to separate directional curvature from full gradient variation. On a binary classification problem, the method achieves the lowest final cross-entropy, objective value, and gradient norm, together with the largest classification margin among the compared scalar gradient methods. On a nonconvex Hölder regression problem, it attains the lowest final objective gap and gradient norm. These results indicate that one-sided Hölder curvature is an effective adaptive step-size signal when full-gradient variation is inflated by directions that do not hinder descent.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。