用在线强化学习优化流匹配模型，提升文本图像生成质量与一致性。
Flow-GRPO: Training Flow Matching Models via Online RL
- 将确定性微分方程转为随机微分方程，支持强化学习探索采样
- 降噪步数减少但推理步数不变，采样效率显著提升
- 在复杂生成和文字渲染任务中表现优异，适合高质量图像生成需求
我们提出Flow-GRPO，首个将在线策略梯度强化学习引入流匹配模型的方法。该方法采用两项关键技术：(1) 将确定性常微分方程（ODE）转换为等效的随机微分方程（SDE），使模型在所有时间步保持原始边际分布，从而支持强化学习中的统计采样探索；(2) 降噪简化策略，在不减少推理步数的前提下降低训练阶段的去噪步骤，大幅提高采样效率且不影响性能。实验表明，Flow-GRPO在多个文本到图像任务中均有效。在组合生成任务中，经强化学习调优的SD3.5-M模型实现近乎完美的物体数量、空间关系及细粒度属性，使GenEval准确率从63%提升至95%；在视觉文本渲染中，准确率从59%升至92%，显著改善文本生成效果。此外，模型在人类偏好对齐方面也有显著提升，且极少出现奖励滥用现象，即奖励提升未伴随图像质量或多样性的明显下降。
原文摘要 · Abstract (English)
We propose Flow-GRPO, the first method to integrate online policy gradient reinforcement learning (RL) into flow matching models. Our approach uses two key strategies: (1) an ODE-to-SDE conversion that transforms a deterministic Ordinary Differential Equation (ODE) into an equivalent Stochastic Differential Equation (SDE) that matches the original model's marginal distribution at all timesteps, enabling statistical sampling for RL exploration; and (2) a Denoising Reduction strategy that reduces training denoising steps while retaining the original number of inference steps, significantly improving sampling efficiency without sacrificing performance. Empirically, Flow-GRPO is effective across multiple text-to-image tasks. For compositional generation, RL-tuned SD3.5-M generates nearly perfect object counts, spatial relations, and fine-grained attributes, increasing GenEval accuracy from $63\%$ to $95\%$. In visual text rendering, accuracy improves from $59\%$ to $92\%$, greatly enhancing text generation. Flow-GRPO also achieves substantial gains in human preference alignment. Notably, very little reward hacking occurred, meaning rewards did not increase at the cost of appreciable image quality or diversity degradation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。