arXiv:2511.22331math.OCcs.AI2025-11被引 3

揭示双层优化中条件数对收敛速度的决定性影响,首次证明理论下界。

On the Condition Number Dependency in Bilevel Optimization

  • 从条件数角度建立双层优化新下界,揭示收敛瓶颈
  • 在下层函数为二次时,下界紧致到对数因子内
  • 适用于非凸-强凸、随机等多场景,指导算法设计

双层优化通过上层目标函数最小化,其可行域由下层问题的解构成。本文研究当上层非凸、下层强凸时,用一阶方法寻找ε-驻点的预言机复杂度。已有工作达到近最优的$ ilde{/mathcal{O}}(ar κ_y^{7/2} ε^{-2})$上界。本文建立新的$Ω(κ_y^{5/2} ε^{-2})$下界,其中$κ_y ≤ ar κ_y$为下层条件数。该下界首次在条件数依赖上揭示双层问题与极小极大问题的差距,且在下层函数为二次时紧致到对数因子内。该下界可扩展至多种情形:(1) 二阶及任意光滑问题,分别得到$Ω(κ_y^{9/4} ε^{-7/4})$和$Ω(κ_y^{13/6} ε^{-5/3})$;(2) 凸-强凸问题,将先前最佳下界$Ω(κ_y / ext{√}ε)$提升至$Ω(κ_y^{3/2} / ext{√}ε)$;(3) 随机非凸-强凸问题,分别给出$Ω(κ_y^4 ε^{-4})$(随机海森向量积)和$Ω(κ_y^{9/2} ε^{-4})$(随机一阶方法)的下界。

原文摘要 · Abstract (English)

Bilevel optimization minimizes an objective function, defined by an upper-level problem whose feasible region is the solution of a lower-level problem. We study the oracle complexity of finding an $ε$-stationary point with first-order methods when the upper-level problem is nonconvex, and the lower-level problem is strongly convex. Recent works achieve a $\tilde{\mathcal{O}}(\bar κ_y^{7/2} ε^{-2})$ upper bound that is near-optimal in $ε$. In this work, we establish a new $Ω(κ_y^{5/2} ε^{-2})$ lower bound, where $κ_y \le \bar κ_y$ is the lower-level condition number. Our lower bound establishes the first provable gap {in terms of condition number dependency} between bilevel problems and minimax problems in this setup, and \textit{is tight up to logarithmic factors when the lower-level function is quadratic.} Our lower bounds can be extended to various settings. (1) For second-order and arbitrarily smooth problems, we show lower bounds of $Ω(κ_y^{9/4} ε^{-7/4})$ and $Ω(κ_y^{13/6} ε^{-5/3})$, respectively. (2) For convex--strongly-convex problems, we improve the previously best lower bound (Ji and Liang, JMLR 2022) from $Ω(κ_y /\sqrtε)$ to $Ω(κ_y^{3/2} / \sqrtε)$. (3) For stochastic nonconvex--strongly-convex problems, we also show the lower bounds of $Ω(κ_y^4 ε^{-4})$ and $Ω(κ_y^{9/2} ε^{-4})$ for stochastic Hessian-vector-product and stochastic first-order methods, respectively.

双层优化条件数下界分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。