用进化策略降低LLM微调内存占用，性能媲美强化学习
ESSAM: A Novel Competitive Evolution Strategies Approach to Reinforcement Learning for Memory Efficient LLMs Fine-Tuning
- 结合进化策略与锐度感知优化，实现全参数微调
- 在GSM8K上达78.27%准确率，内存消耗仅为PPO的1/18
- 适合资源受限场景，尤其适合追求低内存高泛化的研究者
强化学习（RL）已成为提升大语言模型（LLMs）数学推理能力的关键步骤，但其常伴随高昂的GPU内存开销，限制了在资源受限环境中的应用。为此，我们提出进化策略与锐度感知最大化结合的ESSAM框架，通过零阶参数空间搜索与锐度感知优化，提升模型泛化能力。在主流数学推理任务GSM8K上，ESSAM平均准确率达78.27%，性能与主流强化学习方法相当：优于PPO（77.72%），接近GRPO（78.34%），并在部分模型上超越二者。泛化实验显示，使用ESSAM训练的模型在6个数据集中有5个达到最优表现，证明其强泛化能力。在内存方面，相较PPO降低18倍，相较GRPO降低10倍，实现极低内存占用。此外，我们设计加速版ESSAM，速度接近翻倍，内存不变，平均准确率达78.02%，仍优于PPO。
原文摘要 · Abstract (English)
Reinforcement learning (RL) has become a key training step for improving mathematical reasoning in large language models (LLMs), but it often has high GPU memory usage, which makes it hard to use in settings with limited resources. To reduce these issues, we propose Evolution Strategies with Sharpness-Aware Maximization (ESSAM), a full parameter fine-tuning framework that tightly combines the zero-order search in parameter space from Evolution Strategies (ES) with the Sharpness-Aware Maximization (SAM) to improve generalization. We conduct fine-tuning experiments on the mainstream mathematica reasoning task GSM8K. The results show that ESSAM achieves an average accuracy of 78.27\% across all models and its overall performance is comparable to RL methods. It surpasses classic RL algorithm PPO with an accuracy of 77.72\% and is comparable to GRPO with an accuracy of 78.34\%, and even surpassing them on some models. Further generalization experiments show that the models trained with ESSAM exhibit stronger generalization ability. Their average performance achieves the best results on 5 out of 6 datasets, indicating that ESSAM can effectively improve the generalization performance of fine-tuned models. In terms of GPU memory usage, ESSAM reduces the average GPU memory usage by $18\times$ compared to PPO and by $10\times$ compared to GRPO, achieving an extremely low GPU memory usage. In addition, we design an accelerated variant of ESSAM, which achieves nearly a twofold speedup while maintaining the same GPU memory usage as ESSAM, and attains an average accuracy of 78.02\% across all models, outperforming PPO. Code: https://github.com/szs777/ESSAM
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。