提出方向对齐的零阶优化方法，提升梯度估计精度。
Revisiting Zeroth-Order Optimization: Minimum-Variance Two-Point Estimators and Directionally Aligned Perturbations
- 设计方向对齐的随机扰动，动态适应梯度方向。
- 理论证明在小步长下方差更小，实测性能更优。
- 适合需要高精度梯度估计的黑盒优化任务。
本文研究两点式零阶梯度估计器，推导出当扰动步长趋于零时，使估计器渐近方差最小的扰动分布。通过在扰动分布空间上建立约束泛函优化问题，发现最优扰动可与真实梯度方向对齐，而非保持固定长度。现有研究多关注固定长度扰动，忽视了方向对齐的优势。为此，本文深入分析方向对齐扰动（DAP）的理论与实证性质，其能在关键方向自适应提高估计精度。同时，针对使用δ-无偏随机扰动的随机梯度下降，给出了收敛性分析，将现有复杂度界扩展至更广泛的扰动类型。在合成问题与实际任务上的实验表明，DAP在特定条件下显著优于传统方法。
原文摘要 · Abstract (English)
In this paper, we explore the two-point zeroth-order gradient estimator and identify the distribution of random perturbations that minimizes the estimator's asymptotic variance as the perturbation stepsize tends to zero. We formulate it as a constrained functional optimization problem over the space of perturbation distributions. Our findings reveal that such desired perturbations can align directionally with the true gradient, instead of maintaining a fixed length. While existing research has largely focused on fixed-length perturbations, the potential advantages of directional alignment have been overlooked. To address this gap, we delve into the theoretical and empirical properties of the directionally aligned perturbation (DAP) scheme, which adaptively offers higher accuracy along critical directions. Additionally, we provide a convergence analysis for stochastic gradient descent using $δ$-unbiased random perturbations, extending existing complexity bounds to a wider range of perturbations. Through empirical evaluations on both synthetic problems and practical tasks, we demonstrate that DAPs outperform traditional methods under specific conditions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。