让大模型学会自我优化，测试时自动提升答题能力。
Learning to Self-Evolve
- 用强化学习训练模型在测试时迭代优化自身上下文
- 40亿参数模型在两个任务上超越GPT-5和Claude等强基线
- 可迁移指导其他模型，无需重新训练
我们提出Learning to Self-Evolve（LSE），一种强化学习框架，训练大语言模型（LLMs）在测试时自主优化其上下文。该方法置于测试时自演化场景中，模型通过已有问题的反馈迭代改进上下文，以应对新问题。现有方法完全依赖模型内在推理能力，未显式训练自演化能力。LSE将多步演化问题简化为单步强化学习目标，每次上下文修改由下游性能提升奖励。我们结合树状引导的演化循环，在Text-to-SQL生成（BIRD）和通用问答（MMLU-Redux）任务上，40亿参数模型经LSE训练后，优于基于GPT-5和Claude Sonnet 4.5的自演化策略，以及GEPA、TextGrad等提示优化方法，并能无需额外训练地迁移至其他模型。结果表明，将自演化视为可学习技能具有显著有效性。
原文摘要 · Abstract (English)
We introduce Learning to Self-Evolve (LSE), a reinforcement learning framework that trains large language models (LLMs) to improve their own contexts at test time. We situate LSE in the setting of test-time self-evolution, where a model iteratively refines its context from feedback on seen problems to perform better on new ones. Existing approaches rely entirely on the inherent reasoning ability of the model and never explicitly train it for this task. LSE reduces the multi-step evolution problem to a single-step RL objective, where each context edit is rewarded by the improvement in downstream performance. We pair this objective with a tree-guided evolution loop. On Text-to-SQL generation (BIRD) and general question answering (MMLU-Redux), a 4B-parameter model trained with LSE outperforms self-evolving policies powered by GPT-5 and Claude Sonnet 4.5, as well as prompt optimization methods including GEPA and TextGrad, and transfers to guide other models without additional training. Our results highlight the effectiveness of treating self-evolution as a learnable skill.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。