提出新平滑性概念,让近似二阶优化全局快速收敛。
Gradient-Normalized Smoothness for Optimization with Approximate Hessians
- 引入梯度归一化光滑性,连接海森近似与梯度线性化。
- 在凸与非凸问题上实现最优全局收敛率,包括逻辑回归等场景。
- 适用于广义自协调函数,适合研究优化算法的学者参考。
本文提出新型优化算法,结合近似二阶信息与梯度正则化技术,实现凸与非凸目标函数的快速全局收敛。核心创新在于提出「梯度归一化光滑性」这一新概念,刻画当前点附近梯度场的良好相对近似范围。该理论揭示了海森近似与梯度线性化之间的内在联系。梯度归一化光滑性不依赖具体函数类,能将梯度场与海森近似局部信息转化为全局行为。该概念使近似二阶算法获得通用全局收敛保证,自动恢复霍尔德连续海森及三阶导数、拟自协调函数、一阶光滑类等已有最优收敛率,并扩展至广义自协调函数。结果直接应用于逻辑回归、softmax问题中近似海森的全局线性收敛,以及使用费舍尔与高斯-牛顿近似的非凸优化场景。
原文摘要 · Abstract (English)
In this work, we develop new optimization algorithms that use approximate second-order information combined with the gradient regularization technique to achieve fast global convergence rates for both convex and non-convex objectives. The key innovation of our analysis is a novel notion called Gradient-Normalized Smoothness, which characterizes the maximum radius of a ball around the current point that yields a good relative approximation of the gradient field. Our theory establishes a natural intrinsic connection between Hessian approximation and the linearization of the gradient. Importantly, Gradient-Normalized Smoothness does not depend on the specific problem class of the objective functions, while effectively translating local information about the gradient field and Hessian approximation into the global behavior of the method. This new concept equips approximate second-order algorithms with universal global convergence guarantees, recovering state-of-the-art rates for functions with Hölder-continuous Hessians and third derivatives, quasi-self-concordant functions, as well as smooth classes in first-order optimization. These rates are achieved automatically and extend to broader classes, such as generalized self-concordant functions. We demonstrate direct applications of our results for global linear rates in logistic regression and softmax problems with approximate Hessians, as well as in non-convex optimization using Fisher and Gauss-Newton approximations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。