让AI自己优化指令,从执行反馈中学会写更好的操作步骤。
Write, Execute, Refine: From Skill Followers to Skill Optimizers via Reinforcement Learning from Execution Feedback

- 用执行结果反向训练优化器,逐步改进技能生成能力。
- 在多轮任务数据集上比无技能基线提升7.80分,超过现有模型。
- 适合想提升工具调用智能体性能的研究者和开发者。
专家编写的自然语言技能可提升工具使用智能体的表现,但由智能体自动生成的技能表现却比不使用技能时低8-11分。这一差距表明:遵循流程指导与基于执行证据改进技能是两种不同能力。推理阶段的循环能修复技能,但无法提升下一个技能的生成模型。本文研究如何将中间技能的执行经验组织为优化器的训练状态。提出WER(Write, Execute, Refine)多阶段框架,在冻结执行器外训练一个技能优化器。优化器生成技能,冻结智能体重复执行,程序化验证器评估结果并提供相对评分,筛选出混合成功与失败轨迹。匹配的成功与失败轨迹构成下一阶段的优化状态,使优化器从自身输出的后果中学习。在BFCL v4多轮任务和tau2-bench数据集上,WER相比无技能基线分别提升7.80和3.85分;在相同优化流程下,优于未训练优化器的同一骨干模型9.35和10.29分。训练好的4B参数优化器在BFCL v4上达到76.63%的通过率,超越所有测试的现成通用型技能优化模型平均表现。
原文摘要 · Abstract (English)
Expert-written natural language skills can improve tool-using agents, yet agent-authored skills perform 8-11 points worse than using no skill. This gap suggests that following procedural guidance and improving it from execution evidence are distinct capabilities. Inference time loops can repair skills but do not improve the model that writes the next one. We study how to organize execution experience from intermediate skills into training states for an optimizer. We introduce WER (Write, Execute, and Refine), a multi-phase framework that trains a Skill Optimizer outside a frozen executor. The optimizer proposes skills, a frozen agent executes each repeatedly, and a programmatic verifier scores the outcomes. The scores provide relative credit and select mixed-outcome records. Matched successful and failed trajectories from these records form the next phase's refinement states, so the optimizer learns from the consequences of its earlier outputs. On BFCL v4 multi-turn and tau2-bench, WER improves average Pass@1 over the no-skill baseline by 7.80 and 3.85 points, respectively. Under an identical refinement workflow, it outperforms the same backbone without optimizer training by 9.35 and 10.29 points. The trained 4B optimizer reaches 76.63 percent on BFCL v4, outperforming all evaluated off-the-shelf general-purpose models used as skill optimizers on average.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。