arXiv:2605.09359cs.LGcs.AI2026-05被引 6

用强化学习让智能体技能自动进化,提升复杂任务表现。

Skill-R1: Agent Skill Evolution via Reinforcement Learning

论文配图:Skill-R1: Agent Skill Evolution via Reinforcement Learning
图 1 · 摘自论文原文
  • 设计可迭代优化的技能生成器,通过验证奖励反馈改进技能
  • 在多步任务上相比基线提升显著,尤其在复杂场景中表现优异
  • 无需修改主模型,兼容闭源系统,成本远低于微调

智能体大语言模型常依赖可复用的自然语言技能来指导规划与工具使用。当前技能优化多依赖提示工程或对任务模型本身进行对齐,代价高、依赖模型且难以用于闭源模型。技能优化是递归过程,需在当前条件下的行为质量与未来技能改进之间进行双重信用分配。我们提出Skill-R1,一种基于可验证奖励的实例级递归技能优化强化学习框架。该框架不更新任务模型,而是训练一个轻量级技能生成器,根据任务上下文、先前推理结果及其验证反馈生成新技能,引导冻结的任务模型。这保持了对开源与闭源模型的黑箱兼容性,同时大幅降低适应成本。技能优化分多代进行:每轮由当前技能生成推理轨迹,其验证结果反馈以生成下一轮修订。为优化此过程,引入双层组相对策略优化目标,结合同代内比较与跨代进步奖励。前者对比相同技能下的不同轨迹,后者鼓励跨代行为改善。两者共同提供方向性技能演化的原则性目标,而非一次性的自我精炼。实验表明,Skill-R1在具备可验证奖励的基准测试中持续优于无技能基线和标准GRPO,在复杂多步任务上提升尤为明显。

原文摘要 · Abstract (English)

Agentic large language models often rely on skills, reusable natural language procedures that guide planning, action, and tool use. In practice, skills are typically improved through prompt engineering or by aligning the task LLM itself, which is costly, model-specific, and often infeasible for closed-source models. Skill optimization is not a one-step problem but a recurrent process with two coupled levels of credit assignment: a useful skill must improve rollout quality under current conditioning, while a useful revision must turn observed outcomes into a better skill for the next round. We propose Skill-R1, a reinforcement learning framework for instance-level recurrent skill optimization from verifiable rewards. Rather than updating the task LLM, Skill-R1 trains a lightweight skill generator that conditions on the task context, prior rollouts, and their verified outcomes to produce skills that steer a frozen task LLM. This preserves black-box compatibility with both open- and closed-source models while making adaptation substantially cheaper than model-level updates. Skill-R1 proceeds over multiple generations: at each step, the current skill induces rollouts whose verified outcomes are fed back to produce the next revision. To optimize this recurrent process, we introduce a bi-level group-relative policy optimization objective combining intra-generation and inter-generation advantages. The intra-generation term compares rollouts under shared skill conditioning, while the inter-generation term rewards revisions that improve behavior across successive generations. Together, these provide a principled objective for directional skill evolution rather than one-shot self-refinement. Empirically, Skill-R1 achieves consistent gains over no-skill baselines and standard GRPO across benchmarks with verifiable rewards, with particularly strong improvements on complex, multi-step tasks.

智能体强化学习技能进化可验证奖励

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。