提出首个面向流匹配模型的离策略优化方法，显著提升训练效率。
OP-GRPO: Efficient Off-Policy GRPO for Flow-Matching Models
- 通过选择高质量轨迹并动态重用，实现离策略训练
- 在图像和视频生成任务上仅需34.2%训练步数即达相同效果
- 适合追求高效训练且关注生成质量的研究者
基于GRPO的后训练在提升流匹配模型生成质量方面表现卓越，但其基于在线策略的训练范式导致样本效率低下。为此，本文提出首个专为流匹配模型设计的离策略GRPO框架OP-GRPO。首先，主动筛选高质量轨迹并自适应地存入回放缓冲区以供后续迭代复用；其次，为缓解离策略样本带来的分布偏移，提出序列级重要性采样修正方法，在保持GRPO剪裁机制完整性的同时确保策略更新稳定；第三，理论与实证表明晚期去噪步骤会产生病态的离策略比率，因此通过截断晚期轨迹予以缓解。在图像与视频生成基准测试中，OP-GRPO平均仅需34.2%的训练步数即可达到或超越Flow-GRPO性能，大幅提高训练效率，同时维持生成质量。
原文摘要 · Abstract (English)
Post training via GRPO has demonstrated remarkable effectiveness in improving the generation quality of flow-matching models. However, GRPO suffers from inherently low sample efficiency due to its on-policy training paradigm. To address this limitation, we present OP-GRPO, the first Off-Policy GRPO framework tailored for flow-matching models. First, we actively select high-quality trajectories and adaptively incorporate them into a replay buffer for reuse in subsequent training iterations. Second, to mitigate the distribution shift introduced by off-policy samples, we propose a sequence-level importance sampling correction that preserves the integrity of GRPO's clipping mechanism while ensuring stable policy updates. Third, we theoretically and empirically show that late denoising steps yield ill-conditioned off-policy ratios, and mitigate this by truncating trajectories at late steps. Across image and video generation benchmarks, OP-GRPO achieves comparable or superior performance to Flow-GRPO with only 34.2% of the training steps on average, yielding substantial gains in training efficiency while maintaining generation quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。