提出统一凸性下双层优化新算法,突破传统强凸假设限制。
Bilevel Optimization with Lower-Level Uniform Convexity: Theory and Algorithm
- 基于下层均匀凸性设计新算法UniBiO,利用随机梯度与海森向量积信息
- 达到ε-驻点的最优阶复杂度˜O(ε^{-5p+6}),p=2时逼近理论最优
- 适用于超参数优化等场景,尤其在非强凸但具均匀凸性的任务中表现优异
双层优化是机器学习中超参数优化等应用的常见框架,其上层优化受下层问题约束。现有方法通常假设下层函数满足强凸或Polyak-Łojasiewicz条件以保证非渐近收敛至小超梯度解,但这些假设在实际中未必成立。近期研究指出,对一般凸下层函数寻找小超梯度本质上不可行。本文识别出一类介于强凸与一般凸之间的可解类——下层均匀凸性(指数p≥2),建立新的隐式微分定理,刻画了超目标函数的光滑性。基于此,设计了新随机算法UniBiO,依赖提供随机梯度和海森向量积信息的预言机,实现了找到ε-驻点的˜O(ε^{-5p+6}) oracle复杂度。当p=2时,该复杂度在ε依赖上达到最优,仅差对数因子。实验在合成任务和数据超清洗中验证了算法有效性。
原文摘要 · Abstract (English)
Bilevel optimization is a hierarchical framework where an upper-level optimization problem is constrained by a lower-level problem, commonly used in machine learning applications such as hyperparameter optimization. Existing bilevel optimization methods typically assume strong convexity or Polyak-Łojasiewicz (PL) conditions for the lower-level function to establish non-asymptotic convergence to a solution with small hypergradient. However, these assumptions may not hold in practice, and recent work~\citep{chen2024finding} has shown that bilevel optimization is inherently intractable for general convex lower-level functions with the goal of finding small hypergradients. In this paper, we identify a tractable class of bilevel optimization problems that interpolates between lower-level strong convexity and general convexity via \emph{lower-level uniform convexity}. For uniformly convex lower-level functions with exponent $p\geq 2$, we establish a novel implicit differentiation theorem characterizing the hyperobjective's smoothness property. Building on this, we design a new stochastic algorithm, termed UniBiO, with provable convergence guarantees, based on an oracle that provides stochastic gradient and Hessian-vector product information for the bilevel problems. Our algorithm achieves $\widetilde{O}(ε^{-5p+6})$ oracle complexity bound for finding $ε$-stationary points. Notably, our complexity bounds match the optimal rates in terms of the $ε$ dependency for strongly convex lower-level functions ($p=2$), up to logarithmic factors. Our theoretical findings are validated through experiments on synthetic tasks and data hyper-cleaning, demonstrating the effectiveness of our proposed algorithm.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。