通过流程重构提升强化学习训练效率，实现更快更灵活的规模化训练。
RLinf: Flexible and Efficient Large-scale Reinforcement Learning via Macro-to-Micro Flow Transformation
- 将高层RL任务自动分解并重组为优化执行流，提升系统灵活性。
- 在推理与具身强化学习任务上，训练吞吐量提升1.07至2.43倍。
- 适合需要高效大规模训练的AI智能体研发团队使用。
强化学习（RL）在推动通用人工智能、代理智能和具身智能方面展现出巨大潜力，但其工作流固有的异构性和动态性常导致硬件利用率低、训练速度慢。本文提出RLinf，一种高性能强化学习训练系统，核心洞察在于系统灵活性是高效训练的主要瓶颈。基于新颖的宏到微流程转换（M2Flow）设计范式，RLinf可自动在时空维度上分解高层次、易组合的RL工作流，并重新组合为优化执行流。结合RLinf工作者的自适应通信能力，我们设计了上下文切换与弹性流水线以实现流程转换，并采用基于性能分析的调度策略生成最优执行计划。在推理型与具身型强化学习任务上的大量评估表明，RLinf持续优于现有先进系统，在端到端训练吞吐量上实现1.07×–2.43×加速。
原文摘要 · Abstract (English)
Reinforcement learning (RL) has demonstrated immense potential in advancing artificial general intelligence, agentic intelligence, and embodied intelligence. However, the inherent heterogeneity and dynamicity of RL workflows often lead to low hardware utilization and slow training on existing systems. In this paper, we present RLinf, a high-performance RL training system based on our key observation that the major roadblock to efficient RL training lies in system flexibility. To maximize flexibility and efficiency, RLinf is built atop a novel RL system design paradigm called macro-to-micro flow transformation (M2Flow), which automatically breaks down high-level, easy-to-compose RL workflows at both the temporal and spatial dimensions, and recomposes them into optimized execution flows. Supported by RLinf worker's adaptive communication capability, we devise context switching and elastic pipelining to realize M2Flow transformation, and a profiling-guided scheduling policy to generate optimal execution plans. Extensive evaluations on both reasoning RL and embodied RL tasks demonstrate that RLinf consistently outperforms state-of-the-art systems, achieving $1.07\times-2.43\times$ speedup in end-to-end training throughput.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。