arXiv:2512.22288cs.LGcs.AI2025-12被引 6

让扩散模型的训练与生成过程同步优化,提升图像生成质量。

Co-GRPO: Co-Optimized Group Relative Policy Optimization for Masked Diffusion Model

  • 将生成过程建模为联合优化的马尔可夫决策过程,同时训练模型和推理策略。
  • 在四个评测基准上显著提升生成质量,最高达18.7%的指标改进。
  • 无需复杂反向传播,适合希望提升生成效果的研究者和工程师。

最近,掩码扩散模型(MDMs)在视觉、语言及跨模态生成任务中展现出巨大潜力。然而,其训练与推理过程存在显著差异:推理是多步迭代过程,依赖于影响解码路径的调度策略;而训练通常采用简化的单步BERT式目标,一次性预测被掩码的全部内容。这种步骤层面的简化使训练与推理轨迹脱节,导致调度策略从未在训练中被优化。本文提出Co-GRPO,将MDM生成重构为统一的马尔可夫决策过程(MDP),联合优化模型与推理调度策略。通过在轨迹层面应用分组相对策略优化(Group Relative Policy Optimization),Co-GRPO 在共享奖励下协同优化模型参数与调度参数,无需对多步生成过程进行昂贵的反向传播。该整体优化使训练更贴近推理,显著提升生成质量。在ImageReward、HPS、GenEval和DPG-Bench四个基准上的实验验证了方法的有效性。

原文摘要 · Abstract (English)

Recently, Masked Diffusion Models (MDMs) have shown promising potential across vision, language, and cross-modal generation. However, a notable discrepancy exists between their training and inference procedures. In particular, MDM inference is a multi-step, iterative process governed not only by the model itself but also by various schedules that dictate the token-decoding trajectory (e.g., how many tokens to decode at each step). In contrast, MDMs are typically trained using a simplified, single-step BERT-style objective that masks a subset of tokens and predicts all of them simultaneously. This step-level simplification fundamentally disconnects the training paradigm from the trajectory-level nature of inference, leaving the inference schedules never optimized during training. In this paper, we introduce Co-GRPO, which reformulates MDM generation as a unified Markov Decision Process (MDP) that jointly incorporates both the model and the inference schedule. By applying Group Relative Policy Optimization at the trajectory level, Co-GRPO cooperatively optimizes model parameters and schedule parameters under a shared reward, without requiring costly backpropagation through the multi-step generation process. This holistic optimization aligns training with inference more thoroughly and substantially improves generation quality. Empirical results across four benchmarks-ImageReward, HPS, GenEval, and DPG-Bench-demonstrate the effectiveness of our approach. For more details, please refer to our project page: https://co-grpo.github.io/ .

扩散模型生成优化推理调度

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。