arXiv:2606.03892cs.CLcs.AI2026-06

让大模型在真实环境里学会多步调用工具,解决执行失败和废话连篇问题。

Synthesize and Reward -- Reinforcement Learning for Multi-Step Tool Use in Live Environments

论文配图:Synthesize and Reward -- Reinforcement Learning for Multi-Step Tool Use in Live Environments
图 1 · 摘自论文原文
  • 构建20个带状态的工具服务器,支持真实环境训练。
  • 生成的指令基于实际运行状态,避免调用无效工具。
  • 用智能奖励机制抑制冗长调用,提升效率,适合工具使用场景。

训练大模型协调多步工具调用面临三大挑战:真实有状态的执行环境成本高,合成训练数据常脱离服务器实际状态(导致调用失败),基于召回的强化学习奖励会鼓励冗长的调用模式。本文提出PROVE框架,包含三项贡献:(1) 一个包含20个状态化MCP(Model Context Protocol)服务器的库,提供343个工具,支持会话级状态隔离的实时执行强化学习训练;(2) 一种基于状态机的数据合成管道,生成与实时采样服务器状态一致的多轮工具调用轨迹,确保指令引用的实体真实存在;(3) 一个多组件程序化奖励机制,含自适应效率惩罚项,抵消召回奖励带来的冗余倾向。我们在约1.3万条训练样本上,使用GRPO对四个模型(Qwen3-4B、Qwen3-8B、Qwen2.5-7B、Granite-4.1-8B)进行训练。在BFCL Multi-Turn、tau2-bench和T-Eval测试集上,性能分别提升最高+10.2、+6.8和+6.5分,表明该框架在跨模型族的多步工具编排任务中均实现稳定增益。

原文摘要 · Abstract (English)

Training LLMs to orchestrate multi-step tool calls is held back by three coupled obstacles: realistic stateful execution environments are costly to build, synthetic training queries are often detached from the server's actual state (so the generated tool calls fail to execute), and recall-based RL rewards incentivize verbose tool-calling patterns. We present PROVE (Programmatic Rewards On Verified Environments), a framework with three contributions: (1) a library of 20 stateful MCP (Model Context Protocol) servers exposing 343 tools, enabling live-execution RL training with session-scoped state isolation; (2) a state-machine data synthesis pipeline that generates multi-turn tool-call trajectories grounded in live-sampled server state, so generated queries reference entities that actually exist; and (3) a multi-component programmatic reward with an adaptive efficiency penalty that counters the verbosity incentive of recall-based rewards. We train four models (Qwen3-4B, Qwen3-8B, Qwen2.5-7B, Granite-4.1-8B) with GRPO on the resulting ~13K training examples. On BFCL Multi-Turn, tau2-bench, and T-Eval, PROVE yields improvements of up to +10.2, +6.8, and +6.5 points respectively, demonstrating that this framework yields consistent gains on multi-step tool orchestration across two model families.

强化学习工具调用大模型多步推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。