arXiv:2604.09813cs.AI2026-04被引 3

构建可控制、可验证的工具使用数据,让AI在复杂环境中学会正确调用工具。

Controllable and Verifiable Tool-Use Data Synthesis for Agentic Reinforcement Learning

  • 分两阶段生成带验证的工具使用轨迹,确保数据可靠。
  • 引入干扰工具和错误输出,提升模型应对模糊反馈能力。
  • 适合想提升AI工具调用鲁棒性的研究人员使用。

现有合成工具使用数据集多用于离线监督微调,但强化学习(RL)需要支持奖励可计算的可执行环境。我们提出COVERT,一种两阶段流水线:首先通过多级验证的自演化合成生成可靠的工具使用轨迹;其次应用保持原始答案不变的增强策略,系统性提升环境复杂度,包括干扰工具、间接或模糊用户查询、噪声、多格式或错误的工具输出。该设计使标准情况可通过参考匹配自动计算奖励,特殊行为如错误检测则由轻量级裁判辅助验证,支持工具调用策略的强化学习优化。在Qwen2.5-Instruct-14B上,COVERT-RL使BFCL v3整体准确率从56.5提升至59.9,ACEBench从53.0提升至59.3,通用能力基准无明显退化;与SFT结合后进一步达到62.1和61.8,证明其对工具使用鲁棒性的增益具有叠加性。结果表明,保持原答案的合成环境为强化学习提供了一种实用的优化阶段,可补充监督微调,提升模型在模糊与不可靠工具反馈下的表现。

原文摘要 · Abstract (English)

Existing synthetic tool-use corpora are primarily designed for offline supervised fine-tuning, yet reinforcement learning (RL) requires executable environments that support reward-checkable online rollouts. We propose COVERT, a two-stage pipeline that first generates reliable base tool-use trajectories through self-evolving synthesis with multi-level validation, and then applies oracle-preserving augmentations that systematically increase environmental complexity. These augmentations introduce distractor tools, indirect or ambiguous user queries, and noisy, multi-format, or erroneous tool outputs, while strictly preserving oracle tool calls and final answers as ground truth. This design enables automatic reward computation via reference matching for standard cases and lightweight judge-assisted verification for special behaviors such as error detection, supporting RL optimization of tool-calling policies. On Qwen2.5-Instruct-14B, COVERT-RL improves overall accuracy on BFCL v3 from 56.5 to 59.9 and on ACEBench from 53.0 to 59.3, with minimal regressions on general-ability benchmarks; when stacked on SFT, it further reaches 62.1 and 61.8, confirming additive gains. These results suggest that oracle-preserving synthetic environments offer a practical RL refinement stage, complementary to SFT, for improving tool-use robustness under ambiguity and unreliable tool feedback.

强化学习工具使用数据合成智能体

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。