直接优化扩散模型前向过程,实现高效无引导的在线强化学习。
DiffusionNFT: Online Diffusion Reinforcement with Forward Process
- 通过流匹配在前向过程中直接优化扩散模型。
- 1000步内将GenEval得分从0.24提升至0.98,效率是FlowGRPO的25倍。
- 无需采样轨迹和概率估计,兼容任意求解器,适合多奖励模型应用。
在线强化学习在后训练语言模型中至关重要,但其扩展至扩散模型仍因不可计算的似然性而困难。现有方法对反向采样过程离散化以实现类似GRPO的训练,却面临求解器限制、前后不一致及与无分类器引导(CFG)集成复杂等根本缺陷。本文提出扩散负样本感知微调(DiffusionNFT),一种新型在线强化学习范式,通过流匹配直接在前向过程中优化扩散模型。DiffusionNFT通过对比正负样本生成定义隐式策略改进方向,自然将强化信号融入监督学习目标。该框架支持任意黑箱求解器,无需似然估计,仅需干净图像即可完成策略优化。在同等条件下,DiffusionNFT比FlowGRPO效率高出25倍,且无需使用CFG。例如,仅用1000步即可将GenEval得分从0.24提升至0.98,而FlowGRPO需超5000步并依赖额外CFG才能达到0.95。借助多个奖励模型,DiffusionNFT在所有测试基准上显著提升了SD3.5-Medium的表现。
原文摘要 · Abstract (English)
Online reinforcement learning (RL) has been central to post-training language models, but its extension to diffusion models remains challenging due to intractable likelihoods. Recent works discretize the reverse sampling process to enable GRPO-style training, yet they inherit fundamental drawbacks, including solver restrictions, forward-reverse inconsistency, and complicated integration with classifier-free guidance (CFG). We introduce Diffusion Negative-aware FineTuning (DiffusionNFT), a new online RL paradigm that optimizes diffusion models directly on the forward process via flow matching. DiffusionNFT contrasts positive and negative generations to define an implicit policy improvement direction, naturally incorporating reinforcement signals into the supervised learning objective. This formulation enables training with arbitrary black-box solvers, eliminates the need for likelihood estimation, and requires only clean images rather than sampling trajectories for policy optimization. DiffusionNFT is up to $25\times$ more efficient than FlowGRPO in head-to-head comparisons, while being CFG-free. For instance, DiffusionNFT improves the GenEval score from 0.24 to 0.98 within 1k steps, while FlowGRPO achieves 0.95 with over 5k steps and additional CFG employment. By leveraging multiple reward models, DiffusionNFT significantly boosts the performance of SD3.5-Medium in every benchmark tested.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。