通过分块优化提升文本到图像生成的强化学习效果
Principled RL for Flow Matching Emerges from the Chunk-level Policy Optimization
- 将连续步骤合并为区块，改用区块级策略优化
- 在标准基准上相比GRPO提升最高43%性能
- 适合关注生成质量与偏好对齐的研究者
近期基于组相对策略优化（GRPO）的后训练流匹配在文本到图像生成中展现出强劲潜力，但受限于优势估计不准确的问题。本文提出将连续生成步骤聚合为语义连贯的‘区块’，并将策略优化从步骤级转向区块级，有效缓解该问题。基于此，我们提出首个面向后训练流匹配的区块级强化学习方法——组分块策略优化（GCPO）。大量实验表明，GCPO在标准文本到图像基准和偏好对齐任务上均表现优异，相对于GRPO最高提升43%相对性能，凸显了区块级策略优化的前景。代码已开源。
原文摘要 · Abstract (English)
Recent Progress in post-training flow matching for text-to-image (T2I) generation with Group Relative Policy Optimization (GRPO) has demonstrated strong potential. However, it is hindered by a critical limitation: inaccurate advantage attribution. In this work, we argue that aggregating consecutive steps into a coherent 'chunk' and shifting the policy optimization paradigm from GRPO's step level to the chunk level can effectively mitigate the negative impact of this issue. Building on this insight, we propose Group Chunking Policy Optimization (GCPO), the first chunk-level reinforcement learning approach for post-training flow matching. Extensive experiments demonstrate that GCPO achieves superior performance on both standard T2I benchmarks and preference alignment, with up to 43% relative gains over GRPO, highlighting the promise of chunk-level policy optimization. The code is available on https://github.com/xingzhejun/GCPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。