让大模型通过逐步积累经验自我进化,提升推理能力。
Self-Improving Large Language Models via Progressive Experience Evolution

- 分两阶段:先提炼经验,再用经验优化策略
- 在5个数学推理任务上优于现有方法,三类模型均有效
- 适合研究自进化、强化学习与大模型能力提升的读者
大语言模型实现自我改进不仅需要有效的策略优化,还需将短暂交互经验转化为持久能力的机制。现有方法割裂:测试时方法可提取经验但无法内化参数,训练时方法能更新参数却缺乏可迁移经验的积累机制。为此,我们提出统一的后训练框架SPEE(Self-Progressive Experience Evolution),分两步进行:显式经验演化与隐式策略优化。在经验演化阶段,系统从多轮交互轨迹中提炼、验证并逐步演化可迁移经验,形成持续进化的全局经验池,融合成功与失败轨迹,过滤低效经验,抑制单次轨迹的后验合理化偏差;随后通过特权引导的在线策略自蒸馏(OPSD)将经验内化为策略。在策略优化阶段,基于奖励的强化学习利用这些内化先验探索新解法。五个数学推理基准实验表明,SPEE在三种模型规模下均持续优于测试时与训练时的自进化基线。代码已开源。
原文摘要 · Abstract (English)
Large language models (LLMs) capable of self-improvement require not only effective policy optimization, but also a principled mechanism for transforming transient interaction experience into persistent model capabilities. Existing self-improvement paradigms remain fragmented: test-time methods can explicitly extract experience but cannot internalize it into model parameters, whereas training-time optimization methods can update model parameters but lack an explicit mechanism for accumulating transferable experience. Bridging these two paradigms requires a critical intermediate stage that remains underexplored, namely \emph{experience distillation}. To address this gap, we propose \textbf{SPEE} (\textbf{S}elf-\textbf{P}rogressive \textbf{E}xperience \textbf{E}volution), a unified post-training framework that sequentially performs explicit experience evolution followed by implicit policy optimization. During explicit experience evolution, SPEE reflects on trajectories collected from multiple interactions to extract, verify, and progressively evolve transferable experience, which is subsequently internalized into the policy through privilege-guided On-Policy Self-Distillation (OPSD). During implicit policy optimization, reward-driven reinforcement learning leverages these internalized priors to explore novel solution strategies. In the experience evolution stage, a continuously evolving global experience pool consolidates knowledge from both successful and failed trajectories, filters out low-utility experience, and mitigates post-hoc rationalization induced by individual trajectories. Experiments on five mathematical reasoning benchmarks demonstrate that SPEE consistently outperforms both test-time and training-time self-evolution baselines across three model scales. The source code is available at https://github.com/rrrsj/SPEE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。