提出新采样方法，解决流匹配中强化学习的噪声问题
Coefficients-Preserving Sampling for Reinforcement Learning with Flow Matching
- 借鉴DDIM思想，设计系数保持采样法消除噪声
- 使奖励模型更准确，提升流匹配优化器收敛速度与稳定性
- 适合希望改进生成质量的扩散模型研究者
强化学习（RL）最近被用于提升扩散模型和流匹配模型在图像与视频生成中的输出质量及提示对齐能力。将在线RL方法应用于流匹配的关键挑战在于为确定性框架引入随机性，通常通过随机微分方程（SDE）实现。我们发现，基于SDE的采样会在生成图像中引入显著噪声伪影，严重损害奖励学习过程。理论分析表明，该噪声源于推理阶段注入的过量随机性。为此，我们受去噪扩散隐式模型（DDIM）启发，重新设计采样过程，提出系数保持采样（CPS），有效消除此类噪声。这使得奖励建模更精确，从而实现如Flow-GRPO和Dance-GRPO等基于强化学习的优化器更快、更稳定的收敛。代码将发布于https://github.com/IamCreateAI/FlowCPS。
原文摘要 · Abstract (English)
Reinforcement Learning (RL) has recently emerged as a powerful technique for improving image and video generation in Diffusion and Flow Matching models, specifically for enhancing output quality and alignment with prompts. A critical step for applying online RL methods on Flow Matching is the introduction of stochasticity into the deterministic framework, commonly realized by Stochastic Differential Equation (SDE). Our investigation reveals a significant drawback to this approach: SDE-based sampling introduces pronounced noise artifacts in the generated images, which we found to be detrimental to the reward learning process. A rigorous theoretical analysis traces the origin of this noise to an excess of stochasticity injected during inference. To address this, we draw inspiration from Denoising Diffusion Implicit Models (DDIM) to reformulate the sampling process. Our proposed method, Coefficients-Preserving Sampling (CPS), eliminates these noise artifacts. This leads to more accurate reward modeling, ultimately enabling faster and more stable convergence for reinforcement learning-based optimizers like Flow-GRPO and Dance-GRPO. Code will be released at https://github.com/IamCreateAI/FlowCPS
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。