用小规模梯度更新构建方向子空间,让进化策略高效优化大模型推理能力。
Hyper-ES: Effective Evolution Strategies for LLM Reasoning via Descent Direction Merging

- 先用少量梯度更新获取有效下降方向,构建紧凑适应子空间。
- 在子空间内用CMA-ES优化方向组合系数,提升搜索效率。
- 比GRPO-LoRA性能高1%,且减少10%的梯度计算开销。
进化策略(ES)是资源受限的大语言模型(LLM)推理微调的一种有前景替代方案。然而,直接将ES应用于十亿参数级别的LLM效果极差。在高维参数空间中,大多数随机扰动几乎与有用更新方向正交,导致优化不稳定。本文提出Hyper-ES,一种基于子空间的ES框架,避免了全参数搜索下ES的弱点,同时发挥其在低维优化中的优势。Hyper-ES首先进行少量低成本的梯度微调,获得下降方向。尽管单个方向改进有限,但它们的张成空间能捕捉到有用的推理更新。随后,使用CMA-ES优化层间DARE-TIES合并系数,在该子空间内搜索有意义的方向组合,而非盲目探索全模型扰动。我们在三个Qwen2.5-Instruct和DeepSeek-R1-Distill模型上,针对六个数学推理数据集进行了评估。结果表明,Hyper-ES始终比GRPO-LoRA高出1%性能,且所需消耗梯度更新次数减少10%。代码已开源:https://github.com/kuangrepi/Hyper-ES。
原文摘要 · Abstract (English)
Evolution Strategy (ES) is a promising alternative to gradient-based fine-tuning for resource-constrained Large Language Model (LLM) reasoning. However, directly applying ES to billion-parameter LLMs is highly ineffective. In such high-dimensional parameter spaces, most random perturbations are nearly orthogonal to useful update directions, leading to unstable optimization. We propose Hyper-ES, a subspace-based ES framework that avoids the weakness of ES in full-parameter search while exploiting its strength in low-dimensional optimization. Instead of asking ES to discover useful directions from random perturbations in the LLM parameter space, Hyper-ES first performs a small number of inexpensive gradient-based fine-tuning runs to obtain descent directions. Although each direction may provide only a limited improvement on its own, their span forms a compact adaptation subspace that captures useful reasoning updates. Hyper-ES then applies CMA-ES to optimize layer-wise DARE-TIES merging coefficients within this subspace, allowing ES to search over combinations of meaningful descent directions rather than over arbitrary full-model perturbations. We evaluate Hyper-ES on three Qwen2.5-Instruct and DeepSeek-R1-Distill backbones across six mathematical reasoning datasets. Results show that Hyper-ES consistently outperforms GRPO-LoRA by 1% while requiring 10% fewer space-consuming gradient updates. Code at https://github.com/kuangrepi/Hyper-ES.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。