让大模型通过进化优化提示词和权重，提升自主学习能力。
Evolutionary System Prompt Learning for Reinforcement Learning in LLMs
- 用遗传算法进化系统提示词，与强化学习同步优化
- 在复杂任务上成功将成功率从38.8%提升至45.1%
- 适合研究自主智能体、强化学习与提示工程融合的学者
构建能从经验中自主进化的智能体是人工智能的长期目标。当前大语言模型主要通过自我反思更新上下文，或通过强化学习（RL）更新权重来实现自进化。本文提出进化系统提示学习（E-SPL），一种联合优化模型上下文与权重的方法。在每次强化学习迭代中，E-SPL并行采样多个系统提示下的轨迹，并同时对大模型权重应用强化学习更新，对系统提示进行进化更新。系统提示通过突变和交叉两个遗传算子演化，由大模型自我反思驱动；选择依据跨迭代更新的相对性能评分。该方法自然区分了提示中编码的陈述性知识与权重中编码的过程性知识，在推理与代理任务中均表现更优。例如，在从易到难（AIME → BeyondAIME）的泛化设置中，E-SPL将强化学习成功率从38.8%提升至45.1%，优于仅使用反思提示演化的基线（40.0%）。整体结果表明，强化学习与系统提示进化具有深层协同作用，结合两者可显著提升样本效率与泛化能力。代码已开源。
原文摘要 · Abstract (English)
Building agentic systems that can autonomously self-improve from experience is a longstanding goal of AI. Large language models (LLMs) today primarily self-improve via two mechanisms: self-reflection for context updates, and reinforcement learning (RL) for weight updates. In this work, we propose Evolutionary System Prompt Learning (E-SPL), a method for jointly improving model contexts and model weights. In each RL iteration, E-SPL samples trajectories under multiple system prompts in parallel, then jointly applies RL updates to LLM weights and evolutionary updates to system prompts. System prompts evolve via mutation and crossover, two genetic operators driven by LLM self-reflection; selection is based on relative performance ratings updated across RL iterations. E-SPL encourages a natural division between declarative knowledge encoded in prompts and procedural knowledge encoded in weights, resulting in improved performance across reasoning and agentic tasks. For instance, in an easy-to-hard (AIME $\rightarrow$ BeyondAIME) generalization setting, E-SPL improves RL success rate from 38.8% $\rightarrow$ 45.1% while also outperforming reflective prompt evolution (40.0%). Overall, our results demonstrate that RL and system prompt evolution are deeply synergistic, and combining the two yields consistent gains in sample efficiency and generalization. Code: https://github.com/LunjunZhang/E-SPL
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。