用真实API反向生成可验证的工具调用数据,提升大模型工具使用能力。
Firefly: Illuminating Large-Scale Verified Tool-Call Data Generation from Real APIs

- 逆向生成:从真实API调用结果反推任务,确保标签正确
- 构建1000+工具的图结构,采样子有向无环图聚焦有效流程
- 支持离线复现训练,适合需要可靠工具调用数据的研究者
训练工具调用智能体需要大规模带可验证标签的轨迹数据,但现有方法或合成环境与真实API行为不符,或生成任务缺乏真实结果用于验证。我们提出FireFly,一个从真实MCP服务器生成可验证工具调用数据的流水线。核心思想是反转标准合成流程:先让强语言模型在图引导的有向无环图结构中探索真实API,再从观测到的结果反向合成任务,从而保证标签正确性。为应对真实工具空间规模(约1,000个工具),我们构建成对工具图并采样子有向无环图,聚焦语义一致的工作流。针对实时API的环境漂移问题,我们设计检索增强模拟器,缓存所有探索结果并在训练和评估中回放,实现完全离线、可复现的强化学习。该流水线生成了5,144个经验证的任务,覆盖240个服务器和993个工具。使用GRPO在FireFly上训练的40亿参数模型,在保留测试集上达到Claude Sonnet 4.6水平,并在Tau2-Bench、MCPMark和MCP-Atlas等多个基准上表现提升。
原文摘要 · Abstract (English)
Training tool-calling agents requires large-scale trajectory data with verifiable labels, yet existing approaches either synthesize environments that diverge from real API behavior or generate tasks without ground-truth outcomes for verification. We present FireFly, a pipeline for generating verified tool-call data from real-world MCP servers. Our key insight is to invert the standard synthesis pipeline: rather than generating tasks and hoping they are solvable, we first let a strong LLM explore real APIs along graph-guided DAG structures, then synthesize tasks backward from observed outcomes, guaranteeing label correctness by construction. To handle the scale of real-world tool spaces (${\sim}$1,000 tools), we build a pairwise tool graph and sample sub-DAGs to focus exploration on semantically coherent workflows. To address environment drift in live APIs, we construct a retrieval-augmented simulator that caches all exploration results and replays them during training and evaluation, enabling fully offline and reproducible RL. Applying this pipeline yields 5,144 verified tasks spanning 240 servers and 993 tools. A 4B-parameter model trained with GRPO on FireFly matches Claude Sonnet 4.6 on our held-out test set and shows improvements on multiple tool-calling benchmarks including Tau2-Bench, MCPMark, and MCP-Atlas.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。