提出 AdvantageFlow，用优势加权优化流模型前向过程，提升图像生成质量。
AdvantageFlow: Advantage-Weighted Least Squares for RL in Flow Models
- 通过优势加权前向预测损失优化流模型，避免反向过程复杂性。
- 在 Stable Diffusion 3.5 Medium 上优于 Flow-GRPO 和负向感知微调基线。
- 引入滚动策略正则化稳定训练，适合图像生成与强化学习结合场景。
我们提出 AdvantageFlow，一种针对修正流模型的前向过程强化学习算法。不同于优化反向过程的 Flow-GRPO，AdvantageFlow 优化的是优势加权的前向过程预测损失。当优势为负时，该优化问题不稳定且损失非凸。我们通过滚动策略正则化加以稳定，该方法可降低方差，并源于拟合局部奖励提升的目标分布。我们在 Stable Diffusion 3.5 Medium 的图像生成任务上评估了 AdvantageFlow，结果表明其性能优于 Flow-GRPO 以及基于负向感知微调的前沿前向过程强化学习基线。
原文摘要 · Abstract (English)
We introduce AdvantageFlow, a forward-process reinforcement learning algorithm for rectified flow models. Unlike Flow-GRPO, which optimizes the reverse process, we optimize an advantage-weighted forward-process prediction loss. This optimization problem is unstable when advantages are negative and the loss becomes non-convex. We stabilize it by rollout policy regularization, which reduces variance and arises from fitting a local reward-improving target distribution. We evaluate AdvantageFlow on image generation tasks with Stable Diffusion 3.5 Medium. It outperforms both Flow-GRPO and a state-of-the-art forward-process RL baseline based on negative-aware fine-tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。