arXiv:2608.01678cs.LGcs.CL2026-08

用强化学习逐步生成更有效的智能体技能,提升下游任务表现。

Progressive Agent Skill Generation via Reinforcement Learning

  • 将技能生成视为可逐次评估的编辑过程,引入回滚奖励机制。
  • 在CL-Bench和tau2-bench上分别提升3.3和6.7个百分点的成功率。
  • 适合需要自动构建复杂技能的智能体系统研究者使用。

现有技能生成方法多依赖启发式规则或流水线式整合,需针对不同证据源定制设计。基于学习的方法虽具统一性,但因技能缺乏直接相关性或正确性监督信号,仍面临挑战——其价值只能通过是否提升智能体在下游任务中的表现来判断。为此,本文提出Skill-α,一种基于强化学习的渐进式高质量技能生成方法。具体地,将技能生成建模为序列编辑过程,将技能构建分解为可独立评估的编辑操作,并引入新颖的回滚奖励,通过对比原始与编辑后技能在锚定查询下的下游执行效果来评估每步编辑。大量实验表明,无论在文档到技能还是经验到技能场景中,Skill-α均优于基于启发式或流水线的方法。在GPT-4o主工作器下,其平均下游成功率分别在CL-Bench和tau2-bench上比最强基线高出3.3和6.7个百分点。进一步消融实验验证了回滚奖励与渐进生成的重要性。

原文摘要 · Abstract (English)

Existing skill generation methods largely rely on heuristics or pipeline-style consolidation, which must be specially designed for different evidence sources. In contrast, learning-based approaches offer a more unified way to model skill generation across heterogeneous sources. However, learning-based skill generation remains challenging because skills lack a natural supervision signal based on relevance or correctness; their value can largely be determined only by whether they improve the behavior of the agent on downstream tasks. To address this challenge, we propose Skill-$α$, a reinforcement learning method for progressively generating high-quality agent skills. Specifically, we formulate skill generation as a sequential editing process that decomposes skill construction into individually evaluable edits, and introduce a novel rollback reward that evaluates each edit by comparing downstream execution under the original and edited skills on an anchored query. Extensive experiments show that Skill-$α$ generates more effective skills than methods based on heuristics or pipelines in both document-to-skill and experience-to-skill settings. Under the main GPT-4o worker, Skill-$α$ improves average downstream success rates over the strongest skill-generation baseline by 3.3 points on CL-Bench and 6.7 points on tau2-bench. Further ablations validate the importance of rollback reward and progressive generation.

强化学习技能生成智能体渐进式

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。