揭示了双上升优化与增广拉格朗日法的等价关系，为高效训练提供理论保障。
Dual Optimistic Ascent (PI Control) is the Augmented Lagrangian Method in Disguise
- 将双上升优化视为增广拉格朗日法的另一种形式
- 证明其可线性收敛至所有局部解
- 为乐观参数调参提供理论依据，适合约束深度学习研究者
约束优化是强制神经网络满足特定要求的强大框架。这类问题通常通过求解其极小极大拉格朗日形式的梯度方法来处理，但此类方法常出现震荡且难以找到所有局部解。尽管增广拉格朗日法（ALM）能解决这些问题，实践中人们仍更倾向使用标准拉格朗日上的双上升优化（PI控制），因其在经验上表现良好但缺乏正式保证。本文首次建立二者之间的等价关系：标准拉格朗日上的双上升优化等价于增广拉格朗日上的梯度下降-上升。这一发现使我们能够将ALM的稳健理论保证转移到双上升设置中，证明其在单步一阶框架下线性收敛至所有局部解。此外，该等价关系为乐观超参数的调参提供了原则性指导。本工作填补了约束深度学习中双上升方法经验成功与理论基础之间的关键空白。
原文摘要 · Abstract (English)
Constrained optimization is a powerful framework for enforcing requirements on neural networks. These constrained deep learning problems are typically solved using first-order methods on their min-max Lagrangian formulation, but such approaches often suffer from oscillations and can fail to find all local solutions. While the Augmented Lagrangian method (ALM) addresses these issues, practitioners often favor dual optimistic ascent schemes (PI control) on the standard Lagrangian, which perform well empirically but lack formal guarantees. In this paper, we establish a previously unknown equivalence between these approaches: dual optimistic ascent on the Lagrangian is equivalent to gradient descent-ascent on the Augmented Lagrangian. This finding allows us to transfer the robust theoretical guarantees of the ALM to the dual optimistic setting, proving it converges linearly to all local solutions. Furthermore, the equivalence provides principled guidance for tuning the optimism hyper-parameter. Our work closes a critical gap between the empirical success of dual optimistic methods and their theoretical foundation in the single-step, first-order regime commonly used in constrained deep learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。