首次将强化学习稳定高效地应用于离散扩散模型，显著提升生成质量。
UDM-GRPO: Stable and Efficient Group Relative Policy Optimization for Uniform Discrete Diffusion Models
- 以最终干净样本为动作，获得更精准的优化信号
- 通过前向扩散过程重构轨迹，对齐预训练分布
- 在图像生成和文字识别上表现优异，适合生成任务研究者
统一离散扩散模型（UDM）作为离散生成建模的新范式，但其与强化学习的结合仍待探索。我们发现直接应用GRPO会导致训练不稳定且性能提升有限。为此，提出首个融合UDM与强化学习的框架UDM-GRPO。核心思路包括：(i) 将最终干净样本视为动作，提供更准确稳定的优化信号；(ii) 通过扩散前向过程重构轨迹，使概率路径更契合预训练分布。此外引入降步与无分类器自由生成策略，进一步提升训练效率。UDM-GRPO在多个文本到图像任务中显著提升基线性能：GenEval准确率从69%提升至96%，PickScore从20.46增至23.81，在连续与离散设置下均达领先水平。在OCR基准上，准确率从8%升至57%，验证方法泛化能力。代码已开源。
原文摘要 · Abstract (English)
Uniform Discrete Diffusion Model (UDM) has recently emerged as a promising paradigm for discrete generative modeling; however, its integration with reinforcement learning remains largely unexplored. We observe that naively applying GRPO to UDM leads to training instability and marginal performance gains. To address this, we propose UDM-GRPO, the first framework to integrate UDM with RL. Our method is guided by two key insights: (i) treating the final clean sample as the action provides more accurate and stable optimization signals; and (ii) reconstructing trajectories via the diffusion forward process better aligns probability paths with the pretraining distribution. Additionally, we introduce two strategies, Reduced-Step and CFG-Free, to further improve training efficiency. UDM-GRPO significantly improves base model performance across multiple T2I tasks. Notably, GenEval accuracy improves from $69\%$ to $96\%$ and PickScore increases from $20.46$ to $23.81$, achieving state-of-the-art performance in both continuous and discrete settings. On the OCR benchmark, accuracy rises from $8\%$ to $57\%$, further validating the generalization ability of our method. Code is available at https://github.com/Yovecent/UDM-GRPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。