通过标签调度实现训练与推理分离,消除流水线空洞,提升大规模强化学习效率。
SeamlessFlow: A Trainer Agent Isolation RL Framework Achieving Bubble-Free Pipelines via Tag Scheduling
- 用轨迹管理器解耦训练与复杂代理执行流程,支持中途暂停续跑。
- 动态重分配空闲节点用于推理,消除流水线空洞,GPU利用率显著提升。
- 适合多智能体、长时序等复杂强化学习任务,兼顾稳定与性能。
我们提出 SeamlessFlow,一种基于服务器的强化学习(RL)框架,解决工业级RL中的两大核心挑战:(1) 将强化学习训练与复杂的智能体执行流程解耦;(2) 在保持大规模部署所需的稳定性与可扩展性的前提下,最大化GPU利用率并最小化空闲时间。首先,SeamlessFlow 引入数据平面,将RL训练器与多样且复杂的代理实现解耦,同时维持高吞吐量。一个中心化的轨迹管理器保存完整的交互历史,并支持部分回放,允许回放过程在权重更新期间暂停并无缝恢复,使智能体对服务中断无感知。其次,我们提出基于标签的调度范式,将硬件抽象为带能力标签的资源,统一共置与分离架构。基于此,SeamlessFlow 设计了时空复用流水线,可在训练-推理分离架构中动态将空闲训练节点重新分配至推理任务,彻底消除流水线空洞,充分挖掘异构集群资源。通过融合这些创新,SeamlessFlow 实现了稳定性与高性能的统一,适用于多智能体、长时程等复杂强化学习任务。
原文摘要 · Abstract (English)
We introduce SeamlessFlow, a server based reinforcement learning (RL) framework that addresses two core challenges in industrial scale RL: (1) decoupling RL training from the complex execution flow of agents; (2) maximizing GPU utilization with minimal idle time while preserving the stability and scalability required for large-scale deployments. First, SeamlessFlow introduces a data plane that decouples the RL trainer from diverse, complex agent implementations while sustaining high throughput. A central trajectory manager maintains complete interaction histories and supports partial rollout, allowing rollout to pause for weight updates and resume seamlessly, keeping agents unaware of service interruptions. Second, we propose a tag driven scheduling paradigm that abstracts hardware into capability tagged resources, unifying colocated and disaggregated architectures. Based on this, SeamlessFlow introduces a spatiotemporal multiplexing pipeline that dynamically reassigns idle training nodes to rollout in a train rollout separated setup, eliminating pipeline bubbles and fully exploiting heterogeneous cluster resources. By combining these innovations, SeamlessFlow delivers both stability and high performance, making it well suited for multi agent, long horizon, and other complex RL tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。