用JS散度平衡生成质量与多样性，提升自回归文生图性能
Balancing Performance and Diversity in GRPO Autoregressive Text-to-Image Post-Training
- 在统一f散度框架下分析参考策略差异对优化的影响
- JS散度在多数指标上表现最优，同时保持良好生成多样性
- 适合关注文生图质量与多样性的研究者和开发者
自回归文生图近期发展迅速，但生成图像与人类偏好对齐仍具挑战。基于GRPO的在线强化学习提供有效框架，但现有方法通常将参考策略差异视为固定值，忽视其对策略优化的直接影响。本文在统一的f散度框架中研究该因素，涵盖前向KL、反向KL和JS散度，用于GRPO风格的自回归文生图对齐。系统性理论分析表明，不同散度以不同方式重塑词元级更新。尤其在采样词元塑造形式下，JS正则化通过缓解相对于参考策略的均匀偏差，同时抑制过大偏离，实现良好权衡。在LlamaGen和Janus-7B上的大量实验显示，JS散度在多数评估指标上达到最强或极具竞争力的优化性能，同时保持良好生成多样性。代码已公开于https://github.com/tuoyou-hao/BPD-GRPO。
原文摘要 · Abstract (English)
Autoregressive text-to-image (T2I) generation has recently advanced rapidly, yet aligning generated images with human preferences remains challenging. GRPO-style online reinforcement learning provides an effective framework; however, existing methods typically treat reference-policy divergence as fixed, despite its direct impact on policy optimization. We study this overlooked factor within a unified f-divergence framework, encompassing forward KL, reverse KL, and JS divergence, for GRPO-style autoregressive T2I alignment. Our systematic theoretical analysis reveals that different divergences reshape token-level updates in distinct ways. In particular, under the sampled-token shaping form used, JS regularization achieves a favorable trade-off by mitigating uniform bias relative to the reference policy while still discouraging large deviations. Extensive experiments on LlamaGen and Janus-7B show that JS divergence achieves the strongest or highly competitive optimization performance on most evaluation metrics while maintaining favorable generation diversity. The code is available at https://github.com/tuoyou-hao/BPD-GRPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。