arXiv:2606.19004cs.DCcs.AI2026-06

用闲置的便宜显卡加速扩散模型强化学习训练,成本降六倍。

Spotlight: Synergizing Seed Exploration and Spot GPUs for DiT RL Post-Training

论文配图:Spotlight: Synergizing Seed Exploration and Spot GPUs for DiT RL Post-Training
图 1 · 摘自论文原文
  • 让探索过程在旧模型权重上运行,利用空闲低价显卡并行计算。
  • 训练速度提升4倍,总成本降低1.4至6.4倍,图像质量更优。
  • 适合大规模扩散模型训练团队,尤其关注成本与效率的工程优化者。

扩散变换器(DiT)的强化学习(RL)后训练极为昂贵,需数千块高端显卡。现有方法从两个方向降本:种子探索通过选择高对比度样本提升收敛性,但增加关键路径开销;抢占式显卡(Spot GPUs)可降低成本69%–77%,但因DiT推理结果几乎同时完成,难以实现类似大模型的流水线调度,且预占中断会破坏序列并行(SP)组结构,导致资源碎片化。本文提出Spotlight,首个用于DiT RL后训练的抢占式显卡整合系统。基于两大洞察:(1) 探索过程可容忍旧模型权重,因前一迭代权重仍保持随机种子间的相对排序,使探索可在训练期间于空闲的抢占式显卡上运行;(2) 通过复用节点内状态,可将SP组恢复时间从数分钟缩短至亚秒级。据此,Spotlight引入三项技术:基于博弈论的探索调度器,最大化训练预算内的奖励方差;弹性序列并行机制,借助持久化调度器与节点内权重复制动态重构SP组;预占感知的拉取式请求调度器,平衡负载并在预占时提交在途状态。我们在开源平台ROLL上实现Spotlight,并在Qwen-Image后训练任务上评估。结果表明,Spotlight相较基线实现4倍更快的验证得分达到速度,总成本降低1.4–6.4倍,在512×512和1280×1280分辨率下,于DeepSeek-OCR与Geneval数据集均取得更优图像质量。

原文摘要 · Abstract (English)

Reinforcement learning (RL) post-training of Diffusion Transformers (DiTs) is prohibitively expensive, requiring thousands of high-end GPUs. Existing works explore two directions to reduce cost: seed exploration improves training convergence by selecting high-contrast samples, yet adds compute to the critical path; spot GPUs offer 69--77\% lower cost, yet sit idle during training because DiT rollouts finish nearly simultaneously, which prevents LLM-style pipelining of rollout with training. Spot preemptions further break Sequence Parallelism (SP) groups, fragmenting GPU topology. We present Spotlight, the first system that harvests spot GPUs for DiT RL post-training. Spotlight rests on two key insights we devise: (1)~we show that exploration can tolerate stale model weights because exploration that uses the model weights from the previous iteration preserves the relative ranking of random seeds, allowing exploration to run on idle spot GPUs during training. (2)~SP reconfiguration can reuse on-node state, reducing group recovery from minutes to sub-second launches. Built on these insights, Spotlight introduces three techniques: a bandit-based exploration planner that maximizes reward variance within the training time budget, elastic sequence parallelism that reconfigures SP groups on the fly via persistent schedulers and intra-node weight copying, and a preemption-aware pull-based request scheduler that balances load and commits in-flight state upon preemption. We implement Spotlight on the open-source RL platform ROLL and evaluate it on Qwen-Image post-training. Spotlight reaches the same target validation score $4\times$ faster than baselines, reducing total cost by $1.4$-$6.4\times$ while achieving superior image quality on DeepSeek-OCR and Geneval datasets with resolution $512\times512$ and $1280\times1280$.

强化学习扩散模型低成本训练异构计算

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。