提出新方法解决大量复杂约束下的优化难题，效率显著提升。
Gradient-Free Method for Heavily Constrained Nonconvex Optimization
- 采用双重随机化与动量机制，避免全量约束查询和复杂投影。
- 理论证明可收敛至ε-驻点，实验显示训练更快更准。
- 适合黑箱/白箱约束极多的非凸优化问题，如模型鲁棒性设计。
零阶（ZO）方法在梯度难以获取时表现优异。近年来，针对有约束问题，已有不少基于Frank-Wolfe或投影的ZO方法。然而，在许多实际应用中，存在大量非凸黑箱/白箱约束，现有方法因需查询所有约束函数值并投影到复杂可行集而效率极低甚至失效。本文为解决此类问题，提出一种带动量和自适应步长的双重随机零阶梯度方法（DSZOG）。理论上，我们证明了DSZOG可收敛至约束问题的ε-驻点。在两个应用场景中的实验结果表明，该方法在训练时间和精度上均优于其他约束型零阶方法。
原文摘要 · Abstract (English)
Zeroth-order (ZO) method has been shown to be a powerful method for solving the optimization problem where explicit expression of the gradients is difficult or infeasible to obtain. Recently, due to the practical value of the constrained problems, a lot of ZO Frank-Wolfe or projected ZO methods have been proposed. However, in many applications, we may have a very large number of nonconvex white/black-box constraints, which makes the existing zeroth-order methods extremely inefficient (or even not working) since they need to inquire function value of all the constraints and project the solution to the complicated feasible set. In this paper, to solve the nonconvex problem with a large number of white/black-box constraints, we proposed a doubly stochastic zeroth-order gradient method (DSZOG) with momentum method and adaptive step size. Theoretically, we prove DSZOG can converge to the $ε$-stationary point of the constrained problem. Experimental results in two applications demonstrate the superiority of our method in terms of training time and accuracy compared with other ZO methods for the constrained problem.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。