证明了学习梯度下降参数的优化器训练可收敛，突破了此前缺乏理论支撑的瓶颈。
Learning Provably Improves the Convergence of Gradient Descent
- 基于神经正切核理论，证明了学习梯度下降超参数的L2O模型在二次规划中的训练收敛性。
- 提出确定性初始化策略，使训练在长周期优化中更稳定，避免梯度爆炸。
- 在合成数据上比标准梯度下降提升50%以上最优性，优于现有先进L2O方法。
学习优化（Learn to Optimize, L2O）通过训练基于深度神经网络的优化求解器，在加速凸问题和改善非凸解方面取得成功。然而，现有分析常依赖不切实际的假设，导致对L2O自身训练收敛性的理论支持不足，本文通过实证揭示了这一差距。为此，本文首次基于神经正切核（NTK）理论，证明了学习梯度下降（GD）超参数的L2O模型在二次规划问题中的训练收敛性。同时提出一种确定性初始化策略，有效抑制梯度爆炸，支持长时间优化过程中的稳定训练。实验表明，所提框架在合成数据集上实现超过50%的最优性提升，并展现出优于当前先进L2O方法的鲁棒性。代码已开源：https://github.com/NetX-lab/MathL2OProof-Official。
原文摘要 · Abstract (English)
Learn to Optimize (L2O) trains deep neural network-based solvers for optimization, achieving success in accelerating convex problems and improving non-convex solutions. However, L2O lacks rigorous theoretical backing for its own training convergence, as existing analyses often use unrealistic assumptions -- a gap this work highlights empirically. We bridge this gap by proving the training convergence of L2O models that learn Gradient Descent (GD) hyperparameters for quadratic programming, leveraging the Neural Tangent Kernel (NTK) theory. We propose a deterministic initialization strategy to support our theoretical results and promote stable training over extended optimization horizons by mitigating gradient explosion. Our L2O framework demonstrates over 50% better optimality than GD and superior robustness over state-of-the-art L2O methods on synthetic datasets. The code of our method can be found from https://github.com/NetX-lab/MathL2OProof-Official.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。