arXiv:2603.06623cs.LG2026-03综述被引 2

Flow-GRPO让生成模型更好理解人类偏好,提升输出质量。

Advances in GRPO for Generation Models: A Survey

  • 将GRPO方法扩展到生成模型,实现稳定强化学习对齐。
  • 支持文本到图像、视频、3D等多种生成任务的优化。
  • 适合研究生成模型对齐与强化学习的学者和工程师。

大规模流匹配模型在文本到图像、视频、3D建模和语音合成等生成任务中表现优异,但其输出与人类偏好及任务目标对齐仍具挑战。Flow-GRPO将组相对策略优化(GRPO)扩展至生成模型,实现了生成系统的稳定强化学习对齐。自提出以来,该方法推动了快速研究进展,涵盖方法改进与多领域应用。本文系统梳理了Flow-GRPO及其后续发展,从两个维度展开:一是原始框架外的方法演进,包括奖励信号设计、信用分配、采样效率、多样性保持、奖励欺骗防范及奖励模型构建;二是基于GRPO的对齐在文本到图像、视频生成、图像编辑、语音音频、3D建模、具身视觉语言动作系统、统一多模态模型、自回归与掩码扩散模型及修复任务中的扩展。通过整合理论洞察与实践适配,本综述揭示Flow-GRPO作为现代生成模型通用对齐框架的潜力,并指明可扩展、鲁棒的强化生成未来关键挑战。

原文摘要 · Abstract (English)

Large-scale flow matching models have achieved strong performance across generative tasks such as text-to-image, video, 3D, and speech synthesis. However, aligning their outputs with human preferences and task-specific objectives remains challenging. Flow-GRPO extends Group Relative Policy Optimization (GRPO) to generation models, enabling stable reinforcement learning alignment for generative systems. Since its introduction, Flow-GRPO has triggered rapid research growth, spanning methodological refinements and diverse application domains. This survey provides a comprehensive review of Flow-GRPO and its subsequent developments. We organize existing work along two primary dimensions. First, we analyze methodological advances beyond the original framework, including reward signal design, credit assignment, sampling efficiency, diversity preservation, reward hacking mitigation, and reward model construction. Second, we examine extensions of GRPO-based alignment across generative paradigms and modalities, including text-to-image, video generation, image editing, speech and audio, 3D modeling, embodied vision-language-action systems, unified multimodal models, autoregressive and masked diffusion models, and restoration tasks. By synthesizing theoretical insights and practical adaptations, this survey highlights Flow-GRPO as a general alignment framework for modern generative models and outlines key open challenges for scalable and robust reinforcement-based generation.

生成模型强化学习对齐技术流匹配

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。