让扩散模型的训练与生成过程同步优化,提升图像生成质量。
Co-GRPO: Co-Optimized Group Relative Policy Optimization for Masked Diffusion Model
- 将生成过程建模为联合优化的马尔可夫决策过程,同时训练模型和推理策略。
- 在四个评测基准上显著提升生成质量,最高达18.7%的指标改进。
- 无需复杂反向传播,适合希望提升生成效果的研究者和工程师。
最近,掩码扩散模型(MDMs)在视觉、语言及跨模态生成任务中展现出巨大潜力。然而,其训练与推理过程存在显著差异:推理是多步迭代过程,依赖于影响解码路径的调度策略;而训练通常采用简化的单步BERT式目标,一次性预测被掩码的全部内容。这种步骤层面的简化使训练与推理轨迹脱节,导致调度策略从未在训练中被优化。本文提出Co-GRPO,将MDM生成重构为统一的马尔可夫决策过程(MDP),联合优化模型与推理调度策略。通过在轨迹层面应用分组相对策略优化(Group Relative Policy Optimization),Co-GRPO 在共享奖励下协同优化模型参数与调度参数,无需对多步生成过程进行昂贵的反向传播。该整体优化使训练更贴近推理,显著提升生成质量。在ImageReward、HPS、GenEval和DPG-Bench四个基准上的实验验证了方法的有效性。
原文摘要 · Abstract (English)
Recently, Masked Diffusion Models (MDMs) have shown promising potential across vision, language, and cross-modal generation. However, a notable discrepancy exists between their training and inference procedures. In particular, MDM inference is a multi-step, iterative process governed not only by the model itself but also by various schedules that dictate the token-decoding trajectory (e.g., how many tokens to decode at each step). In contrast, MDMs are typically trained using a simplified, single-step BERT-style objective that masks a subset of tokens and predicts all of them simultaneously. This step-level simplification fundamentally disconnects the training paradigm from the trajectory-level nature of inference, leaving the inference schedules never optimized during training. In this paper, we introduce Co-GRPO, which reformulates MDM generation as a unified Markov Decision Process (MDP) that jointly incorporates both the model and the inference schedule. By applying Group Relative Policy Optimization at the trajectory level, Co-GRPO cooperatively optimizes model parameters and schedule parameters under a shared reward, without requiring costly backpropagation through the multi-step generation process. This holistic optimization aligns training with inference more thoroughly and substantially improves generation quality. Empirical results across four benchmarks-ImageReward, HPS, GenEval, and DPG-Bench-demonstrate the effectiveness of our approach. For more details, please refer to our project page: https://co-grpo.github.io/ .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。