解决图像生成强化学习中的反馈不稳定问题，提升训练速度与图像质量。
PCPO: Proportionate Credit Policy Optimization for Aligning Image Generation Models
- 通过比例化信用分配重构目标函数，稳定梯度更新
- 训练收敛速度更快，图像质量显著优于现有方法
- 适合需要高质量图像生成的强化学习研究者
尽管强化学习已推动文本到图像（T2I）模型对齐进展，但当前最先进的策略梯度方法仍受训练不稳定性与高方差困扰，影响收敛速度并降低图像质量。我们分析发现根本原因在于：生成采样器的数学结构导致各时间步间信用分配不均且波动剧烈。为此，提出比例信用策略优化（PCPO），通过稳定的目标重构和时间步的合理重加权，实现比例化的信用分配。该修正使训练过程更加稳定，显著加速收敛并提升图像质量。质量提升源于有效缓解了递归训练中的模型坍缩现象。PCPO在所有指标上均大幅超越现有策略梯度基线，包括最先进的DanceGRPO。代码已开源：https://github.com/jaylee2000/pcpo/。
原文摘要 · Abstract (English)
While reinforcement learning has advanced the alignment of text-to-image (T2I) models, state-of-the-art policy gradient methods are still hampered by training instability and high variance, hindering convergence speed and compromising image quality. Our analysis identifies a key cause of this instability: disproportionate credit assignment, in which the mathematical structure of the generative sampler produces volatile and non-proportional feedback across timesteps. To address this, we introduce Proportionate Credit Policy Optimization (PCPO), a framework that enforces proportional credit assignment through a stable objective reformulation and a principled reweighting of timesteps. This correction stabilizes the training process, leading to significantly accelerated convergence and superior image quality. The improvement in quality is a direct result of mitigating model collapse, a common failure mode in recursive training. PCPO substantially outperforms existing policy gradient baselines on all fronts, including the state-of-the-art DanceGRPO. Code is available at https://github.com/jaylee2000/pcpo/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。