让指令和策略一起进化,让AI推理更聪明。
Agentic Policy Optimization via Instruction-Policy Co-Evolution
- 指令与策略在强化学习中协同优化,动态调整指令以适应策略变化。
- 在多轮检索与推理任务中,性能显著超越静态指令基线。
- 仅增加少量计算开销,就能发现更优的策略引导指令。
基于可验证奖励的强化学习(RLVR)提升了大语言模型的推理能力,使智能体能进行有效的多轮、工具融合推理。传统方法依赖静态且人工设计的指令,但这些指令可能对基础模型不最优,且随策略改进和环境交互而需调整。为此,本文提出INSPO框架,将指令优化纳入强化学习循环,动态维护指令候选池,通过奖励信号自动评估并定期淘汰低效指令。新指令由基于LLM的优化器通过回放缓冲区中的历史经验进行在线反思生成,以适配当前策略。在多轮检索与推理任务上的实验表明,INSPO显著优于依赖静态指令的强基线,发现能引导智能体走向更策略性推理路径的新指令,实现显著性能提升,计算开销仅小幅增加。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Rewards (RLVR) has advanced the reasoning capability of large language models (LLMs), enabling autonomous agents that can conduct effective multi-turn and tool-integrated reasoning. While instructions serve as the primary protocol for defining agents, RLVR typically relies on static and manually designed instructions. However, those instructions may be suboptimal for the base model, and the optimal instruction may change as the agent's policy improves and explores the interaction with the environment. To bridge the gap, we introduce INSPO, a novel Instruction-Policy co-evolution framework that integrates instruction optimization as a dynamic component of the reinforcement learning (RL) loop. INSPO maintains a dynamic population of instruction candidates that are sampled with questions, where reward signals in RL loops are automatically attributed to each instruction, and low performers are periodically pruned. New instructions are generated and verified through an on-policy reflection mechanism, where an LLM-based optimizer analyzes past experience from a replay buffer and evolves more effective strategies given the current policy. We conduct extensive experiments on multi-turn retrieval and reasoning tasks, demonstrating that INSPO substantially outperforms strong baselines relying on static instructions. INSPO discovers innovative instructions that guide the agent toward more strategic reasoning paths, achieving substantial performance gains with only a marginal increase in computational overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。