JigsawRL通过动态调度提升大模型强化学习训练效率，显著提速。
JigsawRL: Assembling RL Pipelines for Efficient LLM Post-Training
- 将强化学习流水线拆解为子阶段图，暴露资源分配不均问题。
- 在4到64张H100/A100 GPU上，吞吐量比Verl高1.85倍，比StreamRL和AReaL高1.54倍。
- 支持异构流水线，适合追求高效训练的大模型研发团队。
我们提出JigsawRL，一种成本高效的框架，探索管道复用作为强化学习并行的新维度。JigsawRL将每个流水线分解为子阶段图，揭示了传统阶段级系统隐藏的阶段内与跨工作者间的不平衡。在此抽象基础上，JigsawRL通过动态资源分配解决复用干扰，通过迁移长尾回溯数据消除碎片化利用率，并将协调问题建模为图调度问题，采用前瞻启发式求解。在4至64张H100/A100 GPU上，针对不同智能体强化学习流水线和模型，JigsawRL在同步强化学习中实现比Verl最高1.85倍的吞吐提升，在异步强化学习中比StreamRL和AReaL高出1.54倍，同时支持异构流水线，仅带来适度延迟开销。
原文摘要 · Abstract (English)
We present JigsawRL, a cost-efficient framework that explores Pipeline Multiplexing as a new dimension of RL parallelism. JigsawRL decomposes each pipeline into a Sub-Stage Graph that exposes the intra-stage and inter-worker imbalance hidden by stage-level systems. On this abstraction, JigsawRL resolves multiplexing interference through dynamic resource allocation, eliminates fragmented utilization by migrating long-tail rollouts across workers, and formulates their coordination as a graph scheduling problem solved with a look-ahead heuristic. On 4-64 H100/A100 GPUs across different agentic RL pipelines and models, JigsawRL achieves up to 1.85x throughput over Verl on synchronous RL, 1.54x over StreamRL and AReaL on asynchronous RL, and supports heterogeneous pipelines with moderate latency trade-off.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。