进化策略首次实现百亿参数大模型全量微调，效果优于强化学习。
Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning
- 用进化策略直接优化大模型全部参数，无需降维
- 在长时序、延迟奖励任务中表现更稳定可靠
- 适合追求鲁棒性与抗奖励欺骗的模型调优场景
为下游任务微调大语言模型（LLMs）是现代AI部署的关键步骤。强化学习（RL）已成为主流微调范式，支撑了众多前沿大模型。相比之下，由于普遍认为进化策略（ES）无法扩展至现代大模型规模，其长期被忽视。本文首次证明，ES可成功应用于百亿参数级别大模型的全参数微调，且无需降维处理。实验表明，ES在高维参数空间中具备有效搜索能力，在多个维度上超越现有RL方法：对长时序与延迟奖励更具容忍度，跨不同基础模型表现更稳健，更不易受奖励劫持影响，训练过程也更稳定。这些结果表明，ES不仅是RL的可行替代方案，更是一种无需反向传播的全新后训练范式，为超越当前基于强化学习的微调路径开辟了新方向。
原文摘要 · Abstract (English)
Fine-tuning large language models (LLMs) for downstream tasks is an essential stage of modern AI deployment. Reinforcement learning (RL) has emerged as the dominant fine-tuning paradigm, underpinning many state-of-the-art LLMs. In contrast, evolution strategies (ES) has largely been overlooked due to the widespread belief that it does not scale to modern model sizes. This paper overturns this assumption by demonstrating the first successful application of ES to full-parameter fine-tuning of LLMs at the billion-parameter scale, without dimensionality reduction. ES can indeed search over extremely high-dimensional parameter spaces and outperform established RL implementations across multiple axes, including improved tolerance to long-horizon and delayed rewards, robustness across diverse base LLMs, reduced susceptibility to reward hacking, and improved training stability. These findings suggest that ES is not merely a viable alternative to RL, but a fundamentally different and powerful backpropagation-free post-training paradigm that opens a new direction for LLM fine-tuning beyond current RL-based approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。