用直接偏好优化加速扩散模型训练，速度提升20倍。
Reinforcing Diffusion Models by Direct Group Preference Optimization
- 跳过传统策略梯度，直接根据样本组间相对偏好学习
- 采用确定性微分方程采样器，训练速度达现有方法20倍
- 适用于需高效对齐的图像生成场景，适合追求训练速度的研究者
尽管强化学习方法如组相对偏好优化（GRPO）显著提升了大语言模型性能，但将其应用于扩散模型仍具挑战。特别是，GRPO需要随机策略，而最高效的扩散采样器基于确定性常微分方程（ODE）。近期工作通过使用低效的随机微分方程（SDE）采样器引入随机性，但依赖模型无关的高斯噪声导致收敛缓慢。为解决这一矛盾，我们提出直接组偏好优化（DGPO），一种全新的在线强化学习算法，完全摒弃策略梯度框架。DGPO直接从组级别偏好中学习，利用组内样本的相对信息。该设计无需低效的随机策略，从而可使用高效的确定性ODE采样器，实现更快训练。大量实验表明，DGPO训练速度比当前最优方法快约20倍，并在域内与域外奖励指标上均取得更优表现。代码已开源：https://github.com/Luo-Yihong/DGPO。
原文摘要 · Abstract (English)
While reinforcement learning methods such as Group Relative Preference Optimization (GRPO) have significantly enhanced Large Language Models, adapting them to diffusion models remains challenging. In particular, GRPO demands a stochastic policy, yet the most cost-effective diffusion samplers are based on deterministic ODEs. Recent work addresses this issue by using inefficient SDE-based samplers to induce stochasticity, but this reliance on model-agnostic Gaussian noise leads to slow convergence. To resolve this conflict, we propose Direct Group Preference Optimization (DGPO), a new online RL algorithm that dispenses with the policy-gradient framework entirely. DGPO learns directly from group-level preferences, which utilize relative information of samples within groups. This design eliminates the need for inefficient stochastic policies, unlocking the use of efficient deterministic ODE samplers and faster training. Extensive results show that DGPO trains around 20 times faster than existing state-of-the-art methods and achieves superior performance on both in-domain and out-of-domain reward metrics. Code is available at https://github.com/Luo-Yihong/DGPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。