VAMO让零阶优化更快收敛，适合资源受限设备部署。
VAMO: Efficient Zeroth-Order Variance Reduction for SGD with Faster Convergence
- 混合使用小批量SGD与全批量零阶梯度，降低计算开销。
- 收敛速度达O(1/T + 1/b)，优于SGD的O(1/√T)。
- 内存占用低于多数一阶方法，适合边缘设备使用。
大规模非凸优化在深度学习中常见，需平衡收敛速度与计算效率。一阶（FO）优化器虽收敛快、泛化好，但因现代模型规模大而带来高计算与内存开销。零阶（ZO）算法通过估计梯度减轻负担，但在高维场景下收敛慢，实用性受限。我们提出VAMO（VAriance-reduced Mixed-gradient Optimizer），一种基于SVRG框架的随机方差缩减方法，结合小批量SGD与全批量ZO梯度。VAMO采用两点式零阶估计器，实现维度无关的收敛率O(1/T + 1/b)，其中T为迭代次数，b为批次大小，超越纯零阶方法的维度依赖瓶颈，并显著优于SGD的O(1/√T)。此外，我们设计多点变体，通过调整估计点数来缓解O(1/b)误差，平衡收敛与成本。重要的是，VAMO在动态内存需求上低于多数一阶基线，特别适合边缘部署。实验涵盖传统神经网络训练与大语言模型微调，验证其不仅优于现有一阶与零阶方法，且内存开销更轻。
原文摘要 · Abstract (English)
Optimizing large-scale nonconvex problems, common in deep learning, demands balancing rapid convergence with computational efficiency. First-order (FO) optimizers, which serve as today's baselines, provide fast convergence and good generalization but often incur high computation and memory costs due to the large size of modern models. Conversely, zeroth-order (ZO) algorithms reduce this burden using estimated gradients, yet their slow convergence in high-dimensional settings limits practicality. We introduce VAMO (VAriance-reduced Mixed-gradient Optimizer), a stochastic variance-reduced method that extends mini-batch SGD with full-batch ZO gradients under an SVRG-style framework. VAMO's hybrid design utilizes a two-point ZO estimator to achieve a dimension-agnostic convergence rate of $\mathcal{O}(1/T + 1/b)$, where $T$ is the number of iterations and $b$ is the batch-size, surpassing the dimension-dependent slowdown of purely ZO methods and significantly improving over SGD's $\mathcal{O}(1/\sqrt{T})$ rate. Additionally, we propose a multi-point variant that mitigates the $O(1/b)$ error by adjusting the number of estimation points to balance convergence and cost. Importantly, VAMO achieves these gains with smaller dynamic memory requirements than many FO baselines, making it particularly attractive for edge deployment. Experiments including traditional neural network training and LLM finetuning confirm that VAMO not only outperforms established FO and ZO methods, but also does so with a light memory footprint.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。