让大模型用工具更高效,自动解决训练中的环境与调度难题。
MCP-Universe RL: A Framework for Training MCP Tool-Use Agents via Reinforcement Learning

- 用MCP协议统一工具接口,无需改代码即可接入训练
- 自动管理环境与流水线,让GPU在等待工具时仍保持忙碌
- 一套框架支持软件、科研等多领域任务,适配性强
强化学习(RL)已成为提升大语言模型工具使用能力的有效方式,但现有框架大多仅做到策略更新。面对新领域时,用户需自行搭建数百个并发轨迹的独立环境,并实现与训练系统的连接,同时需调度运行以确保GPU在长时间多轮对话中持续工作,尤其当工具调用缓慢时更为困难。本文提出MCP-Universe RL(MCP-U RL),一个开源框架,自动承担这两项关键任务。它采用模型上下文协议(MCP)作为环境接口,使已部署为MCP服务的工具可无缝接入训练,无需添加特定于RL的集成代码。该框架构建了两个可复用的核心层:环境编排层,通过可插拔容器后端实现环境的创建、隔离与回收;滚动编排层,采用分阶段流水线设计,使不同轨迹重叠执行,从而在等待工具响应时维持GPU利用率。训练层则通过已有RL后端(如veRL和slime)进行策略更新。仅通过修改任务配置,即可在gpt-oss-20b上训练出软件工程、深度研究及通用工具使用代理,并在三类任务中均获得奖励提升。
原文摘要 · Abstract (English)
Reinforcement learning (RL) has become an effective way to improve the tool-use ability of large language models (LLMs), but most existing RL frameworks stop at the policy update. For every new domain, the user is left with two hard systems problems: standing up an isolated environment for each of hundreds of concurrent trajectories and connecting it to training, and scheduling the rollout so that the GPU stays busy across long, multi-turn episodes that spend much of their time stalled on slow tool calls. We present MCP-Universe RL (MCP-U RL), an open-source framework that takes over both. It uses the Model Context Protocol (MCP) as the interface to the environment, so any tool already exposed as an MCP server plugs into training with no RL-specific integration code. It builds the two missing layers once and reuses them across domains: an environment-orchestration layer that provisions, isolates, and recycles the MCP environments over a pluggable container backend, and a rollout-orchestration layer whose staged pipeline overlaps trajectories to keep the GPU busy while episodes wait on tools. A backend-agnostic training layer then applies the update through an existing RL backend, with veRL and slime integrations. With one configuration, changing only the task specification, we train software-engineering, deep-research, and general tool-use agents on gpt-oss-20b and improve task reward in all three.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。