用分块进化策略让小资源下大模型智能体高效训练
Cooperative Coevolution for Resource-Constrained Agentic LLM Post-Training
- 将参数空间分块，协同进化搜索提升优化效率
- 在有限算力下达成92%的性能恢复，优于传统进化方法
- 适合资源受限场景下的智能体大模型后训练
使用工具的大型语言模型（LLM）智能体生成长序列多轮轨迹，导致基于梯度的后训练内存开销大。进化策略（ES）无需反向传播，可实现全参数内存高效的后训练，并最终达到与基于梯度的强化学习（RL）相当的性能。但在资源受限场景中，通常仅有少量GPU，ES的高GPU小时消耗导致训练时间过长。为此，我们提出合作参数子空间进化策略（CoPES），通过将完整参数空间分解为低维子空间，并协同搜索以提升优化效率。我们在数学任务上对Qwen3.5-4B工具使用智能体进行后训练，并在五个难度不同的基准上评估。在全参数GRPO最佳验证检查点的GPU小时预算下，CoPES恢复了92%的验证准确率增益，而标准ES仅恢复67%；其理论GPU内存需求低于全参数GRPO的八分之一。在所有五个基准的pass@k指标上，CoPES均持续优于标准ES和基于LoRA的GRPO。额外实验进一步证明了CoPES在问答任务上的优势。结果表明，在资源受限条件下，CoPES显著改善了内存需求与训练时间的权衡。代码已开源：https://github.com/MetaronWang/CoPES
原文摘要 · Abstract (English)
Tool-using large language model (LLM) agents produce long, multi-turn trajectories, making gradient-based post-training memory-intensive. Evolution strategies (ES) enable memory-efficient full-parameter post-training without backpropagation and can eventually match the performance of gradient-based reinforcement learning (RL). However, resource-constrained settings typically offer only a few GPUs, so the high GPU-hour requirements of ES translate into prohibitively long training times. To address this, we introduce Cooperative Parameter-subspace Evolution Strategy (CoPES), a cooperative coevolutionary method that decomposes the full parameter space into lower-dimensional subspaces and searches over them cooperatively to improve optimization efficiency. We post-train a Qwen3.5-4B tool-using agent for the math task and evaluate it on five benchmarks of varying difficulty. Under the GPU-hour budget of full-parameter GRPO's best validation checkpoint, CoPES recovers 92% of GRPO's validation-accuracy gain, versus 67% for standard ES, while its theoretical GPU memory requirement is less than one-eighth that of full-parameter GRPO. It consistently outperforms standard ES and LoRA-based GRPO on all evaluated pass@k metrics across the five benchmarks. Additional experiments further show the advantage of CoPES on the question-answering task. These results demonstrate an improved trade-off between memory requirements and training time for agentic LLM post-training under resource constraints. The code is open-sourced in https://github.com/MetaronWang/CoPES
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。