用学习方法优化拟牛顿法的坐标步长，提速最高达4倍。
A Learn-to-Optimize Approach for Coordinate-Wise Step Sizes for Quasi-Newton Methods
- 基于LSTM学习历史轨迹，自动调整每维坐标步长。
- 在多种任务上收敛速度比固定步长快4倍。
- 理论保证稳定，适合复杂优化问题求解者。
步长调节对优化算法的稳定性与效率至关重要。尽管自适应坐标步长在一阶方法中表现优于标量步长，但在二阶方法中仍研究不足且挑战更大。现有方法如超梯度下降和切割平面法在二阶场景下改进有限或存在困难。本文首先在著名的拟牛顿法BFGS框架内进行理论分析，推导出保证收敛与稳定的坐标步长充分条件。在此基础上，提出一种新的学习优化（L2O）方法，采用LSTM网络学习最优步长，利用历史优化轨迹信息，同时满足理论约束。大量实验表明，该方法在多种优化任务中显著优于标量步长和超梯度方法，最快可实现4倍加速。
原文摘要 · Abstract (English)
Tuning step sizes is crucial for the stability and efficiency of optimization algorithms. While adaptive coordinate-wise step sizes have been shown to outperform scalar step size in first-order methods, their use in second-order methods is still under-explored and more challenging. Current approaches, including hypergradient descent and cutting plane methods, offer limited improvements or encounter difficulties in second-order contexts. To address these limitations, we first conduct a theoretical analysis within the Broyden-Fletcher-Goldfarb-Shanno (BFGS) framework, a prominent quasi-Newton method, and derive sufficient conditions for coordinate-wise step sizes that ensure convergence and stability. Building on this theoretical foundation, we introduce a novel learn-to-optimize (L2O) method that employs LSTM-based networks to learn optimal step sizes by leveraging insights from past optimization trajectories, while inherently respecting the derived theoretical guarantees. Extensive experiments demonstrate that our approach achieves substantial improvements over scalar step size methods and hypergradient descent-based method, offering up to 4$\times$ faster convergence across diverse optimization tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。