ROLL 是面向大规模强化学习的高效可扩展库，支持低成本、灵活训练与快速实验。
Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Library
- 单控制器架构简化训练流水线开发
- 支持分布式训练与高效数据传输，实现可扩展性
- 适合研究者快速试验智能体算法和奖励设计
我们提出 ROLL，一个面向大规模强化学习优化的高效、可扩展且用户友好的开源库。ROLL 面向三类用户：追求成本效益与容错的大规模训练技术先锋、需要灵活控制训练流程的开发者，以及希望快速实验的研究人员。其核心由多个模块构成：首先，采用单控制器架构并抽象并行工作器，简化训练管道开发；其次，通过并行策略与数据传输模块实现高效可扩展训练；第三，推出回滚调度器，对回滚阶段每个样本生命周期进行细粒度管理；第四，环境工作器与奖励工作器支持智能体强化学习算法与奖励设计的快速灵活实验；最后，AutoDeviceMapping 允许用户在不同训练阶段灵活分配资源到各模型。
原文摘要 · Abstract (English)
We introduce ROLL, an efficient, scalable, and user-friendly library designed for Reinforcement Learning Optimization for Large-scale Learning. ROLL caters to three primary user groups: tech pioneers aiming for cost-effective, fault-tolerant large-scale training, developers requiring flexible control over training workflows, and researchers seeking agile experimentation. ROLL is built upon several key modules to serve these user groups effectively. First, a single-controller architecture combined with an abstraction of the parallel worker simplifies the development of the training pipeline. Second, the parallel strategy and data transfer modules enable efficient and scalable training. Third, the rollout scheduler offers fine-grained management of each sample's lifecycle during the rollout stage. Fourth, the environment worker and reward worker support rapid and flexible experimentation with agentic RL algorithms and reward designs. Finally, AutoDeviceMapping allows users to assign resources to different models flexibly across various stages.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。