arXiv:2602.06422cs.CV2026-02中稿 · ICML被引 7

解决生成模型奖励稀疏问题,让每一步都获得有效反馈。

Alleviating Sparse Rewards by Modeling Step-Wise and Long-Term Sampling Effects in Flow-Based GRPO

  • 用每步增量奖励替代最终结果奖励,更精准捕捉每一步影响。
  • 识别转折点并赋予长期奖励,捕捉早期动作的延迟效应。
  • 无需调参且高效,适合文本到图像生成任务优化。

将GRPO应用于流匹配模型在文本到图像生成中已证明有效。然而,现有方法通常将基于结果的奖励传递给所有前置去噪步骤,未能区分每步的局部作用。此外,当前群体排名主要比较相同时间步的轨迹,忽略了轨迹内依赖关系,某些早期去噪行为可通过延迟、隐式交互影响后续状态。本文提出TurningPoint-GRPO(TP-GRPO),一种缓解步骤奖励稀疏并显式建模去噪轨迹长程效应的GRPO框架。TP-GRPO实现两项关键创新:(i) 以步骤级增量奖励取代基于结果的奖励,提供密集且步骤感知的学习信号,更准确分离每步去噪动作的“纯粹”影响;(ii) 识别转折点——即奖励趋势发生符号变化的步骤,这些步骤使后续奖励演化与整体轨迹趋势一致,并为其分配聚合的长期奖励以捕获延迟影响。转折点仅通过增量奖励符号变化检测,确保方法高效且无超参数依赖。大量实验表明,TP-GRPO更有效地利用奖励信号,一致提升生成质量。演示代码见https://github.com/YunzeTong/TurningPoint-GRPO。

原文摘要 · Abstract (English)

Deploying GRPO on Flow Matching models has proven effective for text-to-image generation. However, existing paradigms typically propagate an outcome-based reward to all preceding denoising steps without distinguishing the local effect of each step. Moreover, current group-wise ranking mainly compares trajectories at matched timesteps and ignores within-trajectory dependencies, where certain early denoising actions can affect later states via delayed, implicit interactions. We propose TurningPoint-GRPO (TP-GRPO), a GRPO framework that alleviates step-wise reward sparsity and explicitly models long-term effects within the denoising trajectory. TP-GRPO makes two key innovations: (i) it replaces outcome-based rewards with step-level incremental rewards, providing a dense, step-aware learning signal that better isolates each denoising action's "pure" effect, and (ii) it identifies turning points-steps that flip the local reward trend and make subsequent reward evolution consistent with the overall trajectory trend-and assigns these actions an aggregated long-term reward to capture their delayed impact. Turning points are detected solely via sign changes in incremental rewards, making TP-GRPO efficient and hyperparameter-free. Extensive experiments also demonstrate that TP-GRPO exploits reward signals more effectively and consistently improves generation. Demo code is available at https://github.com/YunzeTong/TurningPoint-GRPO.

生成模型强化学习文本到图像

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。