TideRL通过感知就绪状态调度，显著提升大模型强化学习的训练效率。
TideRL: Boosting Agentic RL Goodput with Readiness-Aware Scheduling
- 引入就绪感知弹性调度，动态管理任务批处理与资源分配。
- 相比同步基线，训练吞吐最高提升5.6倍，等待时间减少77.6%。
- 适合大规模多轮智能体强化学习场景，尤其关注训练效率的团队。
大语言模型的强化学习正转向多轮智能体工作负载，其中推理任务需反复暂停于外部环境，恢复时上下文不断增长，完成时间高度可变。在此场景下，训练吞吐（goodput）比显卡占用率更关键：显卡空等和重复预填充计算均为纯开销。本文提出TideRL，一个就绪感知的弹性强化学习系统，包含连续任务批处理（CTB）、资源感知反向演员流水线（RA²P）和弹性资源扩展（ERS）。CTB保留有用推理状态，RA²P根据就绪队列和到达间隔选择解耦流式或共置聚合，ERS利用相同就绪信号在推理与训练间动态迁移计算资源。在纯文本与多模态智能体工作负载上，TideRL相较同步基线提升训练吞吐最高达5.6倍，优于异步基线超33%，同时达到相近任务性能。此外，KV缓存命中率提升1.58倍，单步训练时间最多缩短44.3%，总等待时间减少最高77.6%。
原文摘要 · Abstract (English)
Reinforcement learning (RL) for large language models is moving toward multi-turn agentic workloads, where rollout tasks repeatedly pause for external environments, resume with growing contexts, and finish at highly variable times. In this setting, RL training goodput, measured by training throughput, matters more than raw GPU occupancy: GPU waiting and repeated prefill recomputation are pure overhead. We present TideRL, a readiness-aware elastic RL system with Continuous Task Batching, Resource-Aware Ref-Actor Pipelining, and Elastic Resource Scaling. CTB preserves useful rollout state, $\textrm{RA}^2\textrm{P}$ selects between decoupled streaming and colocated aggregation from the ready backlog and arrival interval, and ERS moves ranks between rollout and training using the same readiness signals. Across text-only and multi-modal agentic workloads, TideRL improves RL training goodput by up to 5.6$\times$ over synchronous baselines and over 33% over asynchronous baselines, while reaching similar task performance. It also improves KV cache hit rate by 1.58$\times$, reduces per-step training time by up to 44.3%, and cuts total waiting time by up to 77.6%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。