用双向进化搜索提升语言模型自我优化能力,突破传统方法局限。
Self-Improving Language Models with Bidirectional Evolutionary Search

- 结合正向演化与反向目标分解,生成更丰富的候选答案
- 在三个开放问题求解基准上超越现有开源框架表现
- 适合追求推理性能提升的研究者和开发者
搜索被证明是自提升语言模型和智能体的有效方法,用于后训练样本生成和推理。然而,主流方法如best-of-N采样和树搜索存在两大根本缺陷:依赖稀疏验证信号,且候选生成主要通过自回归扩展,探索受限于模型高概率区域。为此,我们提出双向进化搜索(BES),将正向候选演化与反向目标分解相结合。正向搜索中,BES通过演化算子重组部分轨迹,生成难以通过单一模型推演获得的候选。反向搜索中,递归分解任务为可验证子目标,提供密集中间反馈以指导正向搜索。理论分析表明,仅靠扩展生成的候选局限于狭窄熵壳内,而演化算子可突破该限制;反向搜索可指数级减少找到正确答案所需样本数。实验显示,在主流后训练算法失效的挑战性任务上,BES实现稳定提升;在三个开放问题求解基准上,推理时表现优于现有开源框架,平均与最佳性能均更优。代码与训练模型见https://github.com/Embodied-Minds-Lab/BES。
原文摘要 · Abstract (English)
Search has been proposed as an effective method for self-improving language models and agentic systems, both for post-training sample generation and for inference. However, widely used methods such as best-of-N sampling and tree search face two fundamental limitations: they are guided by sparse verification signals, and they construct candidates primarily through autoregressive expansion, restricting exploration to regions with substantial model probability mass. To address these, we propose Bidirectional Evolutionary Search (BES), a search framework that couples forward candidate evolution with backward goal decomposition. In the forward search, BES augments standard expansion with evolution operators that recombine partial trajectories to generate candidates that are difficult to obtain from a single model rollout. In the backward search, BES recursively decomposes the original task into checkable subgoals, producing dense intermediate feedback that guides forward search. We provide theoretical motivation showing that candidates generated by expansion-only search are confined to a narrow entropy shell while evolutionary operators can escape it, and that backward search can exponentially reduce the number of required samples to find a correct answer. Experiments show that on challenging post-training tasks where mainstream post-training algorithms fail to improve, BES enables consistent gains, and on three open problem solving benchmarks at inference time, BES outperforms existing open-source frameworks in both average and best-case performance. Code and trained models are available at https://github.com/Embodied-Minds-Lab/BES.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。