用低秩曲率优化大模型微调，更准更快还省内存
Low-Rank Curvature for Zeroth-Order Optimization in LLM Fine-Tuning
- 通过自适应扰动分布改进零阶梯度估计
- 在标准测试中准确率更高，收敛更快，峰值内存降27.3%
- 适合资源受限下大模型微调的高效优化场景
我们提出LOREN，一种面向大语言模型微调的曲率感知零阶优化方法。现有零阶方法依赖随机扰动进行有限差分梯度估计，常面临高方差和次优搜索方向问题。本方法通过：(i) 将梯度预处理重构为自适应估计各向异性扰动分布的问题；(ii) 利用自然进化策略框架，采用低秩块对角预处理器捕捉曲率信息；(iii) 使用REINFORCE留一法（RLOO）梯度估计器降低方差。在标准大模型基准测试中，该方法优于当前最先进零阶方法，实现更高精度与更快收敛，同时相比MeZO-Adam将峰值内存使用减少高达27.3%。
原文摘要 · Abstract (English)
We introduce LOREN, a curvature-aware zeroth-order (ZO) optimization method for fine-tuning large language models (LLMs). Existing ZO methods, which estimate gradients via finite differences using random perturbations, often suffer from high variance and suboptimal search directions. Our approach addresses these challenges by: (i) reformulating the problem of gradient preconditioning as that of adaptively estimating an anisotropic perturbation distribution for gradient estimation, (ii) capturing curvature through a low-rank block diagonal preconditioner using the framework of natural evolution strategies, and (iii) applying a REINFORCE leave-one-out (RLOO) gradient estimator to reduce variance. Experiments on standard LLM benchmarks show that our method outperforms state-of-the-art ZO methods by achieving higher accuracy and faster convergence, while cutting peak memory usage by up to 27.3% compared with MeZO-Adam.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。