将多任务双层优化拓展至更通用的凸性假设,首次建立收敛标准并设计高效算法。
A Tale of Two Problems: Multi-Task Bilevel Learning Meets Equality Constrained Multi-Objective Optimization

- 将多任务双层学习转化为等式约束多目标优化问题
- 提出加权切比雪夫惩罚算法,实现对帕累托最优解的有限步收敛
- 可系统探索帕累托前沿,适用于复杂多任务学习场景
近年来,双层优化(BLO)在机器学习中受到广泛关注,但现有方法多局限于单任务设置,并依赖下层强凸性假设,限制了其在日益复杂的现代机器学习问题中的应用。本文首次在放宽的下层一般凸性(LLGC)假设下,将多任务双层学习(MTBL)拓展至多任务场景。为此,我们将带LLGC的MTBL问题重新建模为等式约束多目标优化(ECMO)问题。由于ECMO尚无文献研究,我们首先建立基于KKT的帕累托平稳性作为算法收敛准则。在此基础上,提出加权切比雪夫(WC)惩罚算法,在确定性和随机设定下均达到$O(ST^{-rac{1}{2}})$的有限时间收敛速率,其中$S$为目标数,$T$为总迭代次数。通过在$S$维单纯形上调节偏好向量,该方法可系统探索帕累托前沿。最终,ECMO的解可直接映射回原MTBL问题的解,打通两个基础优化框架之间的闭环。
原文摘要 · Abstract (English)
In recent years, bilevel optimization (BLO) has attracted significant attention for its broad applications in machine learning. However, most existing works on BLO remain confined to the single-task setting and rely on the lower-level strong convexity assumption, which significantly restricts their applicability to modern machine learning problems of growing complexity. In this paper, we make the first attempt to extend BLO to the multi-task setting under a relaxed lower-level general convexity (LLGC) assumption. To this end, we reformulate the multi-task bilevel learning (MTBL) problem with LLGC into an equality constrained multi-objective optimization (ECMO) problem. However, ECMO itself is a new problem that has not yet been studied in the literature. To address this gap, we first establish a new Karush-Kuhn-Tucker (KKT)-based Pareto stationarity as the convergence criterion for ECMO algorithm design. Based on this foundation, we propose a weighted Chebyshev (WC)-penalty algorithm that achieves a finite-time convergence rate of $O(ST^{-\frac{1}{2})$ to KKT-based Pareto stationarity in both deterministic and stochastic settings, where $S$ denotes the number of objectives, and $T$ is the total iterations. Moreover, by varying the preference vector over the $S$-dimensional simplex, our WC-penalty method systematically explores the Pareto front. Finally, solutions to the ECMO problem translate directly into solutions for the original MTBL problem, thereby closing the loop between these two foundational optimization frameworks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。