提出新方法提升离散流模型生成与理解能力。
dFlowGRPO: Rate-Aware Policy Optimization for Discrete Flow Models

- 统一框架支持多种概率路径与非掩码源分布
- 在图文生成任务上超越现有方法,媲美连续流模型
- 适合研究离散生成模型与强化学习结合的学者
离散流模型(DFMs)是一类灵活的离散数据生成模型,扩散大语言模型(dLLMs)是其特例,具有特定混合路径和掩码源分布。尽管已有研究探索将强化学习用于dLLMs,但对更一般的离散流模型应用仍不足。本文提出离散流GRPO(dFlowGRPO),一种面向离散流模型的统一强化学习框架,支持广泛概率路径与非掩码源分布。我们推导出DFMs的完整轨迹概率,并将去噪过程建模为马尔可夫决策过程,使dFlowGRPO能融合条件转移率与后验模型信息。我们将dFlowGRPO应用于FUDOKI——一种新型多模态离散流模型,在图像生成与多模态理解任务上进行评估。实验表明,dFlowGRPO在文本到图像生成任务上优于现有GRPO类方法,性能媲美使用FlowGRPO训练的连续流模型,同时在理解任务中表现强劲。
原文摘要 · Abstract (English)
Discrete flow models (DFMs) are a class of flexible generative models for generating discrete data, and diffusion large language models (dLLMs) can be viewed as a special case with a specific choice of mixture path and a masked source distribution. While several recent works have explored reinforcement learning into dLLMs, its application to more general discrete flow models remains underexplored. In this work, we present discrete Flow-GRPO (dFlowGRPO), a unified reinforcement learning framework for discrete flow models that supports a broad family of probability paths and non-masked source distributions. We derive the full trajectory probability for DFMs and formulate denoising as a Markov decision process, enabling dFlowGRPO to incorporate information from both the associated conditional transition rates and the posterior model during reinforcement learning. We apply dFlowGRPO to FUDOKI, a recent multimodal discrete flow model, and evaluate it on both image generation and multimodal understanding tasks. Empirical results show that dFlowGRPO outperforms existing GRPO-type methods for dLLMs on text-to-image generation tasks and achieves performance competitive with continuous flow-based models trained using FlowGRPO, while also demonstrating strong capabilities on understanding tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。