arXiv:2608.27351cs.LG2026-08

进化策略让大模型推理更全面,比主流方法覆盖更多解题路径。

Understanding Evolution Strategies for LLM Reasoning: Broader Reasoning Coverage than GRPO

论文配图:Understanding Evolution Strategies for LLM Reasoning: Broader Reasoning Coverage than GRPO
图 1 · 摘自论文原文
  • 用群体进化优化提升模型推理多样性,避免传统方法的熵坍缩。
  • 在高阶通过率(Pass@K)上超越GRPO,同时保持单步通过率(Pass@1)稳定。
  • 适合追求泛化推理能力的研究者,尤其关注高效训练与参数稳定性。

进化策略(ES)作为内存高效的后训练范式,在大语言模型推理中崭露头角。然而其优化机制尚不清晰,难以界定相较于主流方法(如分组相对策略优化,GRPO)的优势范围。本文系统研究了ES的动力学与机制,首次在理论和实证上揭示:相比GRPO,ES能实现更广的推理覆盖,从而更好地挖掘预训练模型的推理潜力。理论上,验证器投影下的群体杰恩-申诺尔多样性有助于提升Pass@K表现;实证上,不同于出现熵坍缩的GRPO,ES在提升Pass@1的同时,显著优于GRPO的Pass@K。进一步提出一种序列式GRPO-ES训练策略,融合两者优势。此外发现,尽管整体参数漂移显著,但任务性能提升仅来自少数大梯度更新,体现功能稀疏性,且外部评估表明未引发灾难性遗忘。最后研究超参数影响,发现大模型下所需群体规模更小。这些发现将ES定位为独立于GRPO的推理后训练范式。

原文摘要 · Abstract (English)

Evolution Strategies (ES) have recently emerged as a memory-efficient post-training paradigm for LLM reasoning. However, the optimization behavior of ES remains understudied, making it hard to define its advantage scope compared to mainstream post-training paradigms (e.g., Group Relative Policy Optimization (GRPO)). By systematically investigating ES dynamics and mechanisms, this paper first identifies a performance advantage of ES over GRPO, theoretically and empirically showing that ES can lead to broader reasoning coverage, thereby better exploiting the reasoning capabilities of pretrained LLMs. Theoretically, we show that verifier-projected Jensen-Shannon diversity across the ES population is helpful to higher Pass@K performances. Empirically, unlike GRPO, which exhibits entropy collapse, ES improves Pass@1 while attaining higher Pass@K than GRPO. We further develop a sequential GRPO-ES training strategy that combines GRPO's strength in Pass@1 with ES's gains in Pass@K. Second, we find that despite substantial whole-model parameter drift, the task-performance gains of ES are only contributed to a sparse subset of larger-magnitude updates. This functional sparsity suggests that large parameter movement need not imply widespread functional change, and held-out evaluations further show that it does not necessarily lead to catastrophic forgetting. Finally, we study how hyperparameter design affects the effectiveness of ES, demonstrating that ES requires a smaller population size in a larger LLM. These findings position ES as a distinct reasoning post-training paradigm rather than a less effective, memory-efficient alternative to GRPO.

进化策略大模型推理后训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。