arXiv:2505.00162cs.LGmath.OC2025-05

用低成本模型加速高成本优化,减少调用次数。

Stochastic Subspace Descent Accelerated via Bi-fidelity Line Search

  • 结合高低精度模型构建代理,优化步长选择更高效。
  • 在多个任务中显著减少高精度函数调用次数,性能更优。
  • 适合高维、高成本计算的黑箱优化问题,如模型微调。

高效优化在众多科学与工程领域仍是核心挑战,尤其当目标函数和梯度评估代价高昂时。尽管零阶优化方法在无法获取梯度时仍具有效性,但其实际表现常受限于函数查询的高成本。本文提出双精度随机子空间下降(BF-SSD)算法,一种新型零阶优化方法,旨在降低计算负担。该方法采用双精度框架,通过组合计算成本低的低精度(LF)和精确的高精度(HF)函数评估构建代理模型,从而实现高效的回溯线搜索以确定步长,并在标准假设下提供理论收敛保证。我们在四个不同问题上进行了全面的实验评估:合成优化基准、双形式核岭回归、对机器学习模型的黑箱对抗攻击,以及基于Transformer的黑箱语言模型微调。数值结果表明,相较于相关基线方法,BF-SSD始终能实现更优的优化性能,同时显著减少所需的高精度函数评估次数。本研究凸显了将双精度策略融入零阶优化的有效性,使BF-SSD成为解决大规模、高维现实应用中优化问题的一种有前景且计算高效的方案。

原文摘要 · Abstract (English)

Efficient optimization remains a fundamental challenge across numerous scientific and engineering domains, especially when objective function and gradient evaluations are computationally expensive. While zeroth-order optimization methods offer effective approaches when gradients are inaccessible, their practical performance can be limited by the high cost associated with function queries. This work introduces the bi-fidelity stochastic subspace descent (BF-SSD) algorithm, a novel zeroth-order optimization method designed to reduce this computational burden. BF-SSD leverages a bi-fidelity framework, constructing a surrogate model from a combination of computationally inexpensive low-fidelity (LF) and accurate high-fidelity (HF) function evaluations. This surrogate model facilitates an efficient backtracking line search for step size selection, for which we provide theoretical convergence guarantees under standard assumptions. We perform a comprehensive empirical evaluation of BF-SSD across four distinct problems: a synthetic optimization benchmark, dual-form kernel ridge regression, black-box adversarial attacks on machine learning models, and transformer-based black-box language model fine-tuning. Numerical results demonstrate that BF-SSD consistently achieves superior optimization performance while requiring significantly fewer HF function evaluations compared to relevant baseline methods. This study highlights the efficacy of integrating bi-fidelity strategies within zeroth-order optimization, positioning BF-SSD as a promising and computationally efficient approach for tackling large-scale, high-dimensional problems encountered in various real-world applications.

零阶优化双精度黑箱优化高效计算

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。