提出DigenRL框架,加速扩散模型的分布式强化学习训练。
Accelerating Disaggregated RL for Visual Generative LLMs with Diffusion-Based Parallelism and Trainer-Assisted Generation
- 采用生成轴流水线与时间步并行,实现推理与训练更细粒度协同。
- 引入弹性训练助手生成机制,动态调用训练显卡辅助生成任务。
- 通过紧耦合异步策略利用流水线尾部空隙,提升资源利用率。
强化学习(RL)已成为主流后训练范式,推动了veRL等高性能自回归大语言模型(LLM)系统的出现。与此同时,基于扩散的RL算法如DanceGRPO和FlowGRPO,正将RL应用从语言推理拓展至扩散式视觉与流形生成。然而,针对扩散生成型大模型的高效强化学习系统仍研究不足。现有方案如veRL-Omni仍依赖资源共置执行,虽简化同步但导致推理与训练资源耦合,限制异构部署与独立扩展。为此,本文提出DigenRL——一种面向扩散生成型大模型的解耦强化学习框架,支持灵活资源配置、兼容异构GPU,并实现高效任务调度。为最大限度减少解耦架构中的执行空洞,我们提出:1)生成轴流水线(GAP)与时间步并行(TSP),在推理与训练间实现更细粒度流水线;2)弹性训练助手生成(TAG)机制,使训练显卡可动态辅助生成任务;3)紧耦合单步约束异步策略,进一步利用流水线尾部空隙。在含16-32张GPU的三组硬件测试平台,使用HunyuanVideo-13B、Wan2.1-14B、FLUX.1-12B和QwenImage-20B等生成模型进行大量实验。结果表明,DigenRL相较当前最优扩散式RL系统veRL-Omni和GenRL,吞吐量提升1.56至2.10倍。
原文摘要 · Abstract (English)
Reinforcement learning (RL) has become a dominant post-training paradigm, driving the emergence of high-performance RL systems such as veRL for autoregressive large language models (LLMs). In parallel, diffusion-oriented RL algorithms, e.g., DanceGRPO and FlowGRPO, have rapidly expanded the scope of RL from language reasoning to diffusion-based visual and flow-based generation. However, efficient RL systems for diffusion generative LLMs remain underexplored. Existing implementations, e.g., veRL-Omni, still rely on colocated execution, which simplifies synchronization but couples rollout and training resources, limits heterogeneous deployment, and constrains independent scaling. To this end, we introduce DigenRL, a disaggregated RL framework for diffusion-based generative LLMs that supports flexible resource allocation, accommodates heterogeneous GPUs, and facilitates efficient task scheduling. To maximally reduce the execution bubbles in the disaggregated architecture, we propose: 1) a generation-axis pipeline (GAP) and time-step parallelism (TSP) in the diffusion architecture to enable finer-grained pipelining between rollout and training; 2) an elastic trainer-assisted generation (TAG) approach to enable the trainer GPU resources to dynamically assist in executing rollout generations; and 3) a tightly one-step constrained asynchronous strategy to further utilize the tail bubble in the pipeline. Extensive experiments are conducted on three hardware testbeds with 16-32 GPUs using HunyuanVideo-13B, Wan2.1-14B, FLUX.1-12B, and QwenImage-20B generative models. Experimental results show that DigenRL achieves 1.56-2.10x throughput improvements over state-of-the-art diffusion RL systems, veRL-Omni and GenRL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。