用可控数据合成与分阶奖励训练大模型完成多步工具调用
Training LLMs for Multi-Step Tool Orchestration with Constrained Data Synthesis and Graduated Rewards
- 构建基于真实API响应缓存的确定性环境，生成复杂度可控的合法调用序列
- 设计分阶奖励机制，分别评估调用正确性和流程依赖一致性，提升部分正确性信号
- 在多个基准上验证技能可迁移，尤其适用于跨生态工具协同场景
多步工具编排对大模型仍是难题，因参数错误常导致完整流程失败。训练面临两大障碍：缺乏支持复杂真实API依赖的环境，以及稀疏二值奖励无法提供部分正确性信号。本文提出一种强化学习框架：首先构建基于大规模真实API响应缓存的确定性环境，实现可控复杂度的合法多步调用序列合成；其次引入分阶奖励，将正确性分解为原子有效性（逐调用粒度的正确性）和编排一致性（依赖关系下的正确排序）。在ComplexFuncBench上，该方法显著提升回合准确率，消融实验表明两个奖励组件均不可或缺。在BFCL v4跨基准评估中，所学编排能力可迁移至完全不同的API生态（如代理式网页搜索、内存管理），持续取得性能提升，同时保持单步性能稳定。代码已开源。
原文摘要 · Abstract (English)
Multi-step tool orchestration remains challenging for LLMs, as state-of-the-art models frequently fail on full sequence execution due to parameter errors. Training for these workflows faces two obstacles: the lack of environments supporting complex real-world API dependencies, and sparse binary rewards that provide no signal for partial correctness. We propose a reinforcement learning framework addressing both challenges. First, we construct a deterministic environment backed by a large-scale cache of real API responses, enabling constrained synthesis of valid multi-step traces with controllable complexity. Second, we introduce a graduated reward that decomposes correctness into atomic validity (call-level correctness at increasing granularity) and orchestration consistency (correct sequencing with dependency respect). On ComplexFuncBench, our approach substantially improves turn accuracy, with ablations confirming both reward components are essential. Cross-benchmark evaluation on BFCL v4 shows that the learned orchestration skills transfer to entirely different API ecosystems (e.g., agentic web search and memory management), yielding consistent gains while maintaining stable single-step performance. Code is available at https://github.com/horizon-rl/ToolOrchestrationReward
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。