arXiv:2605.18703cs.CLcs.LG2026-05被引 6

自动生成可执行环境与自然对话轨迹,提升智能体工具使用能力

EnvFactory: Scaling Tool-Use Agents via Executable Environments Synthesis and Robust RL

论文配图:EnvFactory: Scaling Tool-Use Agents via Executable Environments Synthesis and Robust RL
图 1 · 摘自论文原文
  • 自动构建7个领域85个真实环境,生成多轮自然交互数据
  • 仅用少量环境即实现比之前高5倍的训练效率,性能提升6%-15%
  • 适合研究智能体、强化学习和大模型工具调用的开发者

通过代理强化学习(Agentic RL)赋予大语言模型工具使用能力,受限于可扩展、鲁棒的执行环境不足,以及缺乏体现隐式人类推理的真实训练数据。现有方法依赖昂贵真实API、易幻觉的LLM模拟器或单轮合成环境,且轨迹常过度指定,类似指令序列而非自然意图,降低强化学习训练效果。我们提出EnvFactory,一个全自动化框架,可从真实资源中自主探索并验证有状态的可执行工具环境,并通过拓扑感知采样与校准优化生成具隐式意图的自然多轮轨迹。仅使用7个领域中的85个已验证环境,生成2,575条SFT和强化学习轨迹。尽管环境数量远少于以往工作(常为5倍以上),但训练效率与下游性能显著提升,在Qwen3系列模型上相较基线最高提升+15%(BFCLv3)、+8.6%(MCP-Atlas)、+6%(τ²-Bench与VitaBench)。EnvFactory实现了环境构建与轨迹合成的全自动化,为代理强化学习提供可扩展、可拓展、鲁棒的基础。

原文摘要 · Abstract (English)

Equipping LLMs with tool-use capabilities via Agentic Reinforcement Learning (Agentic RL) is bottlenecked by two challenges: the lack of scalable, robust execution environments and the scarcity of realistic training data that captures implicit human reasoning. Existing approaches depend on costly real-world APIs, hallucination-prone LLM simulators, or synthetic environments that are often single-turn or depend on pre-collected documents. Moreover, synthetic trajectories are frequently over-specified, resembling instruction sequences rather than natural human intents, reducing their effectiveness for RL training. We introduce EnvFactory, a fully automated framework that addresses both challenges. EnvFactory autonomously explores and verifies stateful, executable tool environments from authentic resources, and synthesizes natural multi-turn trajectories through topology-aware sampling and calibrated refinement, producing grounded queries with implicit intents. Using only 85 verified environments across 7 domains, EnvFactory generates 2,575 SFT and RL trajectories. Despite using significantly fewer environments than prior work, which are often 5 times more, EnvFactory achieves superior training efficiency and downstream performance, improving Qwen3-series models by up to +15% on BFCLv3, +8.6% on MCP-Atlas, and +6% on conversational benchmarks including $τ^2$-Bench and VitaBench. By fully automating both environment construction and trajectory synthesis, EnvFactory provides a scalable, extensible, and robust foundation for Agentic RL.

智能体强化学习工具使用数据生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。