用改进的强化学习方法，让图像生成模型更快更稳地对齐人类偏好。
V-GRPO: Online Reinforcement Learning for Denoising Generative Models Is Easier than You Think
- 基于扩散模型的变分代理梯度优化，提升训练稳定性与效率。
- 在文本到图像生成上达到当前最优，速度比MixGRPO快2倍、比DiffusionNFT快3倍。
- 方法简单易实现，适配预训练目标，适合追求高效对齐的生成模型研究者。
将去噪生成模型与人类偏好或可验证奖励对齐仍是关键挑战。尽管策略梯度在线强化学习（RL）提供了一个严谨的后训练框架，但其直接应用受限于这些模型的不可计算似然。先前工作要么在采样轨迹上优化诱导的马尔可夫决策过程（MDP），虽稳定但效率低；要么使用基于扩散证据下界（ELBO）的似然替代，但此前在视觉生成中表现不佳。我们的核心洞察是：基于ELBO的方法实际上可以同时实现稳定与高效。通过降低代理方差并控制梯度步长，我们证明该方法能超越基于MDP的方法。为此，我们提出变分GRPO（V-GRPO），将基于ELBO的代理与组相对策略优化（GRPO）算法结合，并引入一组简单而关键的技术。该方法易于实现，与预训练目标一致，且避免了基于MDP方法的局限。V-GRPO在文本到图像合成任务中达到当前最优性能，相比MixGRPO提速2倍，相比DiffusionNFT提速3倍。
原文摘要 · Abstract (English)
Aligning denoising generative models with human preferences or verifiable rewards remains a key challenge. While policy-gradient online reinforcement learning (RL) offers a principled post-training framework, its direct application is hindered by the intractable likelihoods of these models. Prior work therefore either optimizes an induced Markov decision process (MDP) over sampling trajectories, which is stable but inefficient, or uses likelihood surrogates based on the diffusion evidence lower bound (ELBO), which have so far underperformed on visual generation. Our key insight is that the ELBO-based approach can, in fact, be made both stable and efficient. By reducing surrogate variance and controlling gradient steps, we show that this approach can beat MDP-based methods. To this end, we introduce Variational GRPO (V-GRPO), a method that integrates ELBO-based surrogates with the Group Relative Policy Optimization (GRPO) algorithm, alongside a set of simple yet essential techniques. Our method is easy to implement, aligns with pretraining objectives, and avoids the limitations of MDP-based methods. V-GRPO achieves state-of-the-art performance in text-to-image synthesis, while delivering a $2\times$ speedup over MixGRPO and a $3\times$ speedup over DiffusionNFT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。