为多轮大模型智能体设计可扩展的强化学习滚动服务系统
ProRL Agent: Rollout-as-a-Service for RL Training of Multi-Turn LLM Agents
- 通过API服务化管理智能体全流程滚动，解耦训练与调度
- 在软件工程、数学等任务上实现高效强化学习训练
- 支持无根态高性能计算环境下的多样化任务
多轮大模型智能体在解决复杂交互任务中日益重要，强化学习（RL）是提升其长周期行为的关键。然而，RL训练需要生成大量隔离的滚动轨迹，现有基础设施常将滚动编排与训练循环耦合，导致系统难以迁移和维护。基于滚动即服务理念，我们提出ProRL Agent，一个通过API服务支持智能体全生命周期滚动的可扩展基础设施。ProRL Agent还提供标准化且可扩展的沙箱环境，支持多种智能体任务在无根态高性能计算环境中运行。我们在软件工程、数学、科学、技术与编码任务上验证了ProRL Agent的效能。该系统已开源，并作为NVIDIA NeMo Gym的一部分集成。
原文摘要 · Abstract (English)
Multi-turn LLM agents are increasingly important for solving complex, interactive tasks, and reinforcement learning (RL) is a key ingredient for improving their long-horizon behavior. However, RL training requires generating large numbers of sandboxed rollout trajectories, and existing infrastructures often couple rollout orchestration with the training loop, making systems hard to migrate and maintain. Under the rollout-as-a-service philosophy, we present ProRL Agent , a scalable infrastructure that serves the full agentic rollout lifecycle through an API service. ProRL Agent also provides standardized and extensible sandbox environments that support diverse agentic tasks in rootless HPC settings. We validate ProRL Agent through RL training on software engineering, math, STEM, and coding tasks. ProRL Agent is open-sourced and integrated as part of NVIDIA NeMo Gym.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。