用贝叶斯方法在子空间优化大模型，省显存还更准。
Robust and Efficient Zeroth-Order LLM Fine-Tuning via Adaptive Bayesian Subspace Optimizer
- 在子空间内融合多方向扰动信息，用卡尔曼滤波做贝叶斯推断
- 在OPT-13B上提升6.67%平均性能，低精度下仍稳定
- 适合资源受限场景的高效微调，尤其适合低精度训练
通过函数评估近似梯度的零阶优化可降低大语言模型微调的内存开销。但现有方法本质在单维空间更新，低精度训练时易退化或性能下降。我们提出BSZO，一种自适应贝叶斯子空间零阶优化器，利用卡尔曼滤波在子空间内融合多方向有限差分信息。将每个有限差分测量视为含噪观测，构建子空间投影梯度的后验分布，并通过贝叶斯推断更新，结合残差自适应机制应对噪声变化。理论分析表明，相比标准零阶方法，收敛率提升$ k/γ $倍。在RoBERTa、Mistral和OPT模型上的实验显示，BSZO在各类任务中均优于基线，在OPT-13B上实现高达6.67%的绝对平均性能提升，且在fp16/bf16精度下保持鲁棒性，内存使用仅比仅推理基线高1.00×–1.08×（对比MeZO）。
原文摘要 · Abstract (English)
Fine-tuning large language models (LLMs) with zeroth-order (ZO) optimization reduces memory by approximating gradients through function evaluations. However, existing methods essentially perform updates in a one-dimensional space, and suffer from collapse or substantial performance degradation under low-precision training. We introduce BSZO, an adaptive \textbf{B}ayesian \textbf{S}ubspace \textbf{Z}eroth-Order \textbf{O}ptimizer, which applies Kalman filtering to combine finite-difference information across multiple perturbation directions within a subspace. By treating each finite-difference measurement as a noisy observation, BSZO builds a posterior distribution over the subspace-projected gradient and updates it through Bayesian inference, with a residual-based adaptive mechanism to adapt to noise variations. Theoretical analysis shows that BSZO improves the convergence rate by a factor of $k/γ$ compared to standard ZO methods. Experiments on RoBERTa, Mistral, and OPT models show that BSZO outperforms the baselines across various tasks, achieving up to 6.67\% absolute average improvement on OPT-13B while remaining robust under fp16/bf16 precision and keeping memory usage close to inference-only baselines (1.00$\times$--1.08$\times$ of MeZO).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。