让大模型学会自我迭代优化,提升解决复杂问题的能力。
Teaching LLMs to Self-Evolve: Cultivating Core Meta-Skills with Reinforcement Learning

- 通过强化学习训练模型在代码任务中持续自我改进
- 在7个基准上比最强基线提升10.01%(分布内)和24.12%(分布外)
- 适合研究自主智能、模型自进化与通用问题求解的开发者
通过环境反馈实现测试时的迭代自演化,如AlphaEvolve所示,展现出显著性能提升。我们假设此类演化框架的成功依赖于元技能,例如基于环境反馈的自我反思能力,这种能力能有效支持多轮优化,但传统后训练方法对此关注不足。为此,我们提出MetaEvolve框架,通过数据合成流程、演化感知强化学习(RL)及推理时演化搜索,专门培养这些元技能。具体地,我们在编程任务中构建该框架,利用程序执行提供超越二值正确性的连续奖励信号。基于这些信号,我们合成演化轨迹作为训练数据,包含当前程序、其适应度评分(结合正确性与效率)以及前期尝试历史,并通过可验证的奖励信号(来自测试用例执行)进行强化学习训练。通过大规模代码数据训练,旨在激发可泛化的、领域无关的元技能,使其广泛迁移至缺乏丰富训练信号的开放问题。在7个编码基准上,MetaEvolve在分布内任务上优于最强基线10.01%,在分布外任务上提升24.12%;在完全脱离训练领域的开放算法优化问题上,相对提升达46.9%。结果表明,显式培养自演化元技能为构建更强大、自主演化的智能体提供了原则性路径。
原文摘要 · Abstract (English)
Test-time scaling through iterative self-evolution with environment feedback, as demonstrated by AlphaEvolve, shows remarkable performance gains. We hypothesize that the success of such evolution frameworks hinges on meta-skills, such as self-reflection with environment feedback, that enable effective multi-round refinement, yet are largely neglected by traditional post-training. To bridge this gap, we present MetaEvolve, a framework designed to develop these meta-skills via a data synthesis pipeline, evolution-aware reinforcement learning (RL), and inference-time evolutionary search. Concretely, we ground MetaEvolve in coding, where program execution provides natural, continuous reward signals beyond binary correctness. Building on these signals, we synthesize evolution trajectories as training data, each containing a current program, its fitness score (combining correctness and efficiency), and a history of prior attempts, and train the model via RL with verifiable rewards derived from test case execution. By training on large-scale code data, we aim to inspire generalizable domain-agnostic meta-skills that can transfer broadly to open-ended problems where such rich training signals are scarce. Across seven coding benchmarks, MetaEvolve outperforms the strongest baseline by 10.01% absolute on in-distribution tasks and 24.12% on out-of-distribution tasks. On open-ended algorithm optimization problems entirely outside the training domain, it further achieves a 46.9% relative improvement. These results demonstrate that explicitly cultivating self-evolution meta-skills offers a principled path toward more capable and autonomously self-evolving AI.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。