提出轻量自适应零阶优化器MEAZO，兼顾性能与内存效率
On Adaptivity in Zeroth-Order Optimization
- 仅用单个标量追踪全局步长，实现内存高效的自适应优化
- 在多类大模型上达到与ZO-Adam相当的精度，内存仅需ZO-SGD水平
- 对学习率选择更鲁棒，适合高维、分组结构优化场景
我们研究了自适应零阶（ZO）优化在大语言模型（LLM）微调中的有效性。与以往观点相反，我们发现自适应ZO方法如ZO-Adam相比精心调参的ZO-SGD并无收敛优势，反而带来显著内存开销。分析表明，在高维情况下，ZO梯度缺乏坐标间异质性，导致自适应机制内存低效。基于此，我们提出MEAZO，一种仅追踪单一标量以实现全局步长自适应的内存高效优化器，并在标准假设下提供了理论收敛保证。实验在多个LLM家族和任务中验证：MEAZO性能媲美ZO-Adam，内存占用却与ZO-SGD相当。合成二次问题及LLM微调的额外实验进一步证明，MEAZO在分组或块结构优化设置中对步长选择更具鲁棒性。
原文摘要 · Abstract (English)
We investigate the effectiveness of adaptive zeroth-order (ZO) optimization for memory-constrained fine-tuning of large language models (LLMs). Contrary to prior claims, we show that adaptive ZO methods such as ZO-Adam offer no convergence advantage over well-tuned ZO-SGD, while incurring significant memory overhead. Our analysis reveals that in high dimensions, ZO gradients lack coordinate-wise heterogeneity, rendering adaptive mechanisms memory inefficient. Leveraging this insight, we propose MEAZO, a memory-efficient adaptive ZO optimizer that tracks only a single scalar for global step size adaptation. We support our method with theoretical convergence guarantees under standard assumptions. Experiments across multiple LLM families and tasks demonstrate that MEAZO matches ZO-Adam's performance with the memory footprint of ZO-SGD. Additional experiments on synthetic quadratic problems and LLM fine-tuning further demonstrate MEAZO's enhanced robustness to step size choices, particularly in grouped or block-structured optimization settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。