解决大模型强化学习训练中生成延迟瓶颈,实现高效异步训练。
DORA: A Scalable Asynchronous Reinforcement Learning System for Language Model Training
- 设计多版本流式生成机制,支持多个策略版本并行
- 吞吐量提升2-3倍,大规模训练加速2-4倍
- 兼顾算法正确性,适合工业级大模型训练
强化学习(RL)已成为大语言模型后训练的关键范式,但其生成阶段占总训练时间的50%至80%,因长尾轨迹导致严重性能瓶颈。异步训练可缓解此问题,但带来效率与算法正确性之间的矛盾。本文识别出异步训练中的三大约束:轨迹内策略一致性、数据完整性与有限过时性。现有方法无法从根本上解决长尾轨迹问题,尤其在混合专家模型中更因分布不均而加剧。为此,提出DORA(动态异步生成调度系统),通过算法与系统协同设计,引入多版本流式生成机制,同时维持多个策略版本,实现无气泡延迟且不破坏算法约束。实验表明,DORA在开源基准上吞吐量达当前最优系统2-3倍;在数万加速器规模的工业场景中,相较同步训练提速2-4倍。基于该系统的LongCat-Flash-Thinking模型在复杂推理任务上表现优异,媲美主流先进大模型。
原文摘要 · Abstract (English)
Reinforcement learning (RL) has become a critical paradigm for LLM post-training, yet the rollout phase -- accounting for 50--80% of total step time -- is bottlenecked by skewed generation: long-tailed trajectories indispensable for model performance block the entire training pipeline. Asynchronous training offers a natural remedy by overlapping generation with training, but introduces a fundamental tension between efficiency and algorithmic correctness. We identify three constraints in asynchronous training to preserve convergence: intra-trajectory policy consistency, data integrity, and bounded staleness. Existing approaches fail to intrinsically address the long-tailed trajectory problem, which is further exacerbated by the imbalance characteristic of Mix-of-Experts models, or deviate from the standard RL training formulation, thereby hindering model convergence. Therefore, we propose DORA (Dynamic ORchestration for Asynchronous Rollout), which addresses this challenge through algorithm-system co-design. DORA introduces multi-version streaming rollout, a novel asynchronous paradigm that maintains multiple policy versions concurrently -- simultaneously achieving full bubble elimination without compromising algorithmic constraints. Experimental results demonstrate that our DORA system achieves substantial improvements in throughput -- up to 2--3 times higher than state-of-the-art systems on open-source benchmarks -- without compromising convergence. Furthermore, in large-scale industrial applications with tens of thousands of accelerators, DORA accelerates RL training by 2--4 times compared to synchronous training across various scenarios. The resultant open-source models, LongCat-Flash-Thinking, exhibit competitive performance on complex reasoning benchmarks, matching the capability of most advanced LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。