提出新型零阶优化框架,显著提升高维机器学习任务的查询效率。
An Enhanced Zeroth-Order Stochastic Frank-Wolfe Framework for Constrained Finite-Sum Optimization
- 设计双方差缩减机制,降低零阶采样与有限求和的噪声影响。
- 凸目标下查询复杂度达O(d√n/ε),非凸下为O(d¹·⁵√n/ε²)。
- 无需计算梯度,适合黑箱优化,如对抗攻击与稀疏建模。
我们提出一种增强型零阶随机Frank-Wolfe框架,用于解决大规模机器学习中常见的受限有限求和优化问题。该方法引入新颖的双重方差缩减机制,有效降低零阶代理带来的梯度近似误差以及有限求和目标中的随机采样方差。在凸目标下,算法以O(d√n/ε)的查询复杂度达到ε-次优解;在非凸目标下,复杂度为O(d^{3/2}√n/ε²),且每轮迭代无需计算d个偏导数。这是目前不显式计算梯度的零阶随机Frank-Wolfe算法中最优的复杂度。在稀疏逻辑回归、鲁棒分类及深度网络对抗攻击等任务上的实验验证了该方法在收敛速度与查询效率上的优越性。
原文摘要 · Abstract (English)
We propose an enhanced zeroth-order stochastic Frank-Wolfe framework to address constrained finite-sum optimization problems, a structure prevalent in large-scale machine-learning applications. Our method introduces a novel double variance reduction framework that effectively reduces the gradient approximation variance induced by zeroth-order oracles and the stochastic sampling variance from finite-sum objectives. By leveraging this framework, our algorithm achieves significant improvements in query efficiency, making it particularly well-suited for high-dimensional optimization tasks. Specifically, for convex objectives, the algorithm achieves a query complexity of O(d \sqrt{n}/ε) to find an epsilon-suboptimal solution, where d is the dimensionality and n is the number of functions in the finite-sum objective. For non-convex objectives, it achieves a query complexity of O(d^{3/2}\sqrt{n}/ε^2 ) without requiring the computation ofd partial derivatives at each iteration. These complexities are the best known among zeroth-order stochastic Frank-Wolfe algorithms that avoid explicit gradient calculations. Empirical experiments on convex and non-convex machine learning tasks, including sparse logistic regression, robust classification, and adversarial attacks on deep networks, validate the computational efficiency and scalability of our approach. Our algorithm demonstrates superior performance in both convergence rate and query complexity compared to existing methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。