提出自适应步长随机可行性算法,高效求解复杂约束优化问题。
Randomized Feasibility Methods for Constrained Optimization with Adaptive Step Sizes
- 结合随机采样与Polyak步长,动态调整步长以处理难投影约束。
- 强凸情况下目标值线性收敛,凸非光滑时达到1/√T的最优收敛率。
- 无需调参,适用于机器学习中的公平性约束等实际场景。
针对目标函数在凸函数下水平集交集定义的约束条件下进行最小化的问题,研究了两种情形:(i) 强凸且Lipschitz光滑的目标函数;(ii) 凸但可能不可微的目标函数。为应对难以投影的约束,采用带有Polyak步长的随机可行性算法,并在每轮迭代中随机采样一定数量的约束,同时进行(次)梯度下降以优化目标。在情形(i)中,证明了使用自适应步长时目标函数值期望实现线性收敛至任意容差;在情形(ii)中,提出完全无需调参的自适应步长方案,实现了期望下$O(1/\\/sqrt{T})$的最坏情况收敛率。迭代点的不可行性几乎必然几何衰减,对平均迭代点,建立了关于最优值的期望下界,其依赖于随机采样约束数的分布。在某些采样规模增长策略下,可达到最优收敛速率。在二次约束二次规划(QCQP)、支持向量机(SVM)及具有组公平性约束的逻辑回归问题上的仿真表明,该算法相比现有最优方法具有更高的计算效率。
原文摘要 · Abstract (English)
We consider minimizing an objective function subject to constraints defined by the intersection of lower-level sets of convex functions. We study two cases: (i) strongly convex and Lipschitz-smooth objective function and (ii) convex but possibly nonsmooth objective function. To deal with the constraints that are not easy to project on, we use a randomized feasibility algorithm with Polyak steps and a random number of sampled constraints per iteration, while taking (sub)gradient steps to minimize the objective function. For case (i), we prove linear convergence in expectation of the objective function values to any prescribed tolerance using an adaptive stepsize. For case (ii), we develop a fully problem parameter-free and adaptive stepsize scheme that yields an $O(1/\sqrt{T})$ worst-case rate in expectation. The infeasibility of the iterates decreases geometrically with the number of feasibility updates almost surely, while for the averaged iterates, we establish an expected lower bound on the function values relative to the optimal value that depends on the distribution for the random number of sampled constraints. For certain choices of sample-size growth, optimal rates are achieved. Finally, simulations on a Quadratically Constrained Quadratic Programming (QCQP) problem, Support Vector Machines (SVM), and logistic regression with group fairness constraints demonstrate the computational efficiency of our algorithm compared to other state-of-the-art methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。