通过集群级调度，让多个RLVR任务共享大模型资源，大幅减少空等时间。
PlexRL: Cluster-Level Orchestration of Serviceized LLM Execution for RLVR
- 在集群层面统一调度多个RLVR任务的模型执行，动态填充空闲时段。
- 实测可降低用户GPU小时成本最高达37.58%，提升集群利用率。
- 适合需要高效训练推理模型的研究者和团队，尤其关注成本优化。
强化学习结合可验证奖励（RLVR）最近激发了大语言模型（LLM）强大的推理能力，推动了新算法与数据的快速探索。然而，RLVR训练效率极低：长尾式推理、工具导致的卡顿，以及推理与训练间不对称的资源需求，造成大量无法通过同步流水线、异步推理或共置执行消除的闲置时间。我们指出这种低效是结构性的——单个任务内的空闲不可避免，但跨任务间空闲具有强反相关性，可在集群层面被利用。为此，我们提出PlexRL，一个面向多任务统一LLM服务的集群级运行时。通过集中管理模型部署、状态迁移与函数级调度，在严格亲和性约束下，将不同任务的LLM执行按时间片复用，填满闲置期，且无需昂贵的模型迁移。实现与评估表明，PlexRL显著提升有效集群容量，最高降低用户GPU小时成本37.58%，同时保持算法灵活性，每任务开销极小。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards (RLVR) has recently unlocked strong reasoning capabilities in large language models (LLMs), triggering rapid exploration of new algorithms and data. However, RLVR training is notoriously inefficient: long-tailed rollouts, tool-induced stalls, and asymmetric resource requirements between rollout and training introduce substantial idle time that cannot be eliminated by job-local optimizations such as synchronous pipelining, asynchronous rollout, or colocated execution. We argue that this inefficiency is structural. While idle gaps are unavoidable within individual RLVR jobs, they are largely anti-correlated across jobs and therefore exploitable at the cluster level. Leveraging this observation, we present PlexRL, a cluster-level runtime for multiplexing unified LLM services across RLVR jobs. By centrally managing model placement, state transitions, and function-level scheduling under strict affinity constraints, PlexRL time-slices LLM execution across jobs to fill otherwise idle periods without expensive model migration. Our implementation and evaluations demonstrate that PlexRL significantly improves effective cluster capacity and reduces user GPU hour cost by maximum 37.58% while preserving algorithmic flexibility and introducing minimal per-job overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。