构建大规模工具集成环境,提升大模型长时序任务推理能力
ToolVerse: Unlocking Massive Environments and Long-Horizon Tasks for Agentic Reinforcement Learning

- 基于400个真实协议自动构建含4500工具的训练环境
- 设计依赖图生成长周期任务,使模型在动态环境中表现更优
- 提出细粒度奖励机制,解决长任务中信用分配难题
尽管大型语言模型代理在小型、定义明确的场景中展现出强大推理能力,但在需要无缝工具整合的大规模、多样化和动态的真实世界环境中,其鲁棒性和有效性仍不足。为弥补这一差距,我们提出ToolVerse,一个全面的框架,可扩展代理强化学习环境,并支持代理在工具集成推理(TIR)任务中执行复杂长周期推理。首先,ToolVerse从近400个包含约4500个工具的真实模型上下文协议(MCPs)中自动生成大规模可执行的代理训练环境。其次,我们提出一种基于工具依赖图的任务设计策略,利用动态解锁采样算法生成长周期任务,构建GUST(图解锁采样任务)数据集。第三,为缓解长周期代理强化学习中的信用分配问题,我们提出细粒度的逐轮感知相对优势算法。我们在ToolVerse上进行广泛的代理强化学习训练,并在多个代理基准上评估该框架。实验结果表明,该框架显著提升了大模型在长周期工具使用中的能力,实现明显性能提升,并展现出在动态环境中的稳健推理能力。
原文摘要 · Abstract (English)
While LLM agents demonstrate strong reasoning abilities in compact and well-defined scenarios, they struggle to maintain robustness and effectiveness when faced with large-scale, diverse, and dynamic real-world environments that demand seamless tool integration. To address this gap, we introduce ToolVerse, a comprehensive framework that scales up agentic RL environments and enables agents to perform complex long-horizon reasoning in Tool-Integrated Reasoning (TIR) tasks. First, ToolVerse automatically builds the massive executable agent training environments from nearly 400 real-world Model Context Protocols (MCPs) that contain about 4500 tools. Second, we propose a task design strategy based on a tool dependency graph, utilizing Dynamic Unlocking Sampling Algorithm to generate long-horizon tasks, and produce GUST (Graph Unlocking Sampling Tasks) dataset. Third, to alleviate the credit assigment problem in long-horizon agentic RL, we propose a fine-grained Turn-Aware Relative Advantage algorithm. We conduct extensive Agentic RL training using ToolVerse and evaluate our framework on serveral agentic benchmarks. Experimental results demonstrate that our framework significantly strengthens LLMs' capabilities in long-horizon tool use, achieving a marked performance boost and showcasing robust reasoning within dynamic environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。