提出无偏零阶优化梯度估计器，提升优化精度与收敛速度。
On the Optimal Construction of Unbiased Gradient Estimators for Zeroth-Order Optimization
- 基于函数值构造无偏梯度估计，通过可调分布采样消除偏差
- 理论证明使用该估计的SGD在平滑非凸目标上达到最优复杂度
- 适用于梯度不可得或计算昂贵场景，如语言模型微调
零阶优化（ZOO）是梯度不可得或计算代价高昂时的重要随机优化框架。现有方法普遍因梯度估计存在偏差而受限，除非扰动步长趋于零。本文提出一种仅依赖函数值评估的新颖无偏梯度估计器家族，通过将方向导数重构为望远镜级数，并从精心设计的分布中采样，实现了无偏性且保持良好方差。我们分析其理论性质，推导出四种具体构造的最优缩放分布与扰动步长，并证明使用该估计器的随机梯度下降（SGD）在平滑非凸目标上达到最优复杂度。合成任务与语言模型微调实验表明，相比标准方法，本方法在准确性和收敛性上均有显著提升。
原文摘要 · Abstract (English)
Zeroth-order optimization (ZOO) is an important framework for stochastic optimization when gradients are unavailable or expensive to compute. A potential limitation of existing ZOO methods is the bias inherent in most gradient estimators unless the perturbation stepsize vanishes. In this paper, we overcome this biasedness issue by proposing a novel family of unbiased gradient estimators based solely on function evaluations. By reformulating directional derivatives as a telescoping series and sampling from carefully designed distributions, we construct estimators that eliminate bias while maintaining favorable variance. We analyze their theoretical properties, derive optimal scaling distributions and perturbation stepsizes of four specific constructions, and prove that SGD using the proposed estimators achieves optimal complexity for smooth non-convex objectives. Experiments on synthetic tasks and language model fine-tuning confirm the superior accuracy and convergence of our approach compared to standard methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。