用强化学习动态调整采样组大小，提升流模型训练效率与生成质量
SuperFlow: Training Flow Matching Models with RL on the Fly
- 根据提示词重要性动态调整采样组大小，避免固定分组带来的效率浪费
- 采用连续时间流机制计算每步奖励，解决轨迹奖励误用导致的信用分配偏差
- 训练步骤减少5.4%~56.3%，速度提升5.2%~16.7%，生成效果优于SD3.5-M和Flow-GRPO
基于流的生成模型与强化学习（RL）的进展提升了文本-图像对齐与视觉质量。然而，当前流模型的强化学习训练仍存在两大问题：(i) GRPO风格的固定每提示词分组大小忽略了提示词间采样重要性的差异，导致采样效率低下和训练缓慢；(ii) 轨迹级优势被重复用作每步估计，造成沿流路径的信用分配偏差。我们提出SuperFlow，一种针对流模型的强化学习训练框架，通过方差感知采样动态调整组大小，并基于连续时间流动力学一致地计算每步优势。实验表明，SuperFlow在仅使用原训练步骤5.4%至56.3%的情况下，实现5.2%至16.7%的训练时间缩减，且无需任何架构修改。在标准文本到图像任务中，包括文本渲染、组合图像生成和人类偏好对齐，SuperFlow在性能上超越SD3.5-M达4.6%至47.2%，超越Flow-GRPO达1.7%至16.0%。
原文摘要 · Abstract (English)
Recent progress in flow-based generative models and reinforcement learning (RL) has improved text-image alignment and visual quality. However, current RL training for flow models still has two main problems: (i) GRPO-style fixed per-prompt group sizes ignore variation in sampling importance across prompts, which leads to inefficient sampling and slower training; and (ii) trajectory-level advantages are reused as per-step estimates, which biases credit assignment along the flow. We propose SuperFlow, an RL training framework for flow-based models that adjusts group sizes with variance-aware sampling and computes step-level advantages in a way that is consistent with continuous-time flow dynamics. Empirically, SuperFlow reaches promising performance while using only 5.4% to 56.3% of the original training steps and reduces training time by 5.2% to 16.7% without any architectural changes. On standard text-to-image (T2I) tasks, including text rendering, compositional image generation, and human preference alignment, SuperFlow improves over SD3.5-M by 4.6% to 47.2%, and over Flow-GRPO by 1.7% to 16.0%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。