对比DPO与GRPO在图像生成中的表现，发现奖励模型影响算法泛化能力。
Delving into RL for Image Generation with CoT: A Study on DPO vs. GRPO
- 用DPO和GRPO优化自回归图像生成的思维链推理过程
- 强泛化奖励模型可显著提升算法跨域性能
- 揭示两种算法在文本-图像一致性和美学质量上的差异
近期进展表明强化学习（RL）在提升大语言模型（LLMs）链式思维（CoT）能力中起关键作用。两大主流算法——直接偏好优化（DPO）与组相对策略优化（GRPO）——各有优劣。自回归图像生成可视为一种序列化的CoT推理过程，面临独特挑战：确保文本-图像一致性、提升图像美学质量、设计复杂奖励模型，而非依赖简单规则奖励。尽管已有研究将RL拓展至该领域，但通常缺乏对领域特异性挑战及不同RL策略特性的深入分析。为此，本文首次系统研究了GRPO与DPO在自回归图像生成中的表现，评估其域内性能与域外泛化能力，并考察不同奖励模型对其能力的影响。结果表明，两者各具优势，且具备更强内在泛化能力的奖励模型能有效提升所用RL算法的泛化潜力。此外，我们系统探索三种常见扩展策略，为每种范式提供了高效提升域内与域外性能的独特见解。希望本研究为未来开发更高效的RL算法以实现鲁棒的自回归图像生成链式思维推理提供新路径。代码已开源：https://github.com/ZiyuGuo99/Image-Generation-CoT
原文摘要 · Abstract (English)
Recent advancements underscore the significant role of Reinforcement Learning (RL) in enhancing the Chain-of-Thought (CoT) reasoning capabilities of large language models (LLMs). Two prominent RL algorithms, Direct Preference Optimization (DPO) and Group Relative Policy Optimization (GRPO), are central to these developments, showcasing different pros and cons. Autoregressive image generation, also interpretable as a sequential CoT reasoning process, presents unique challenges distinct from LLM-based CoT reasoning. These encompass ensuring text-image consistency, improving image aesthetic quality, and designing sophisticated reward models, rather than relying on simpler rule-based rewards. While recent efforts have extended RL to this domain, these explorations typically lack an in-depth analysis of the domain-specific challenges and the characteristics of different RL strategies. To bridge this gap, we provide the first comprehensive investigation of the GRPO and DPO algorithms in autoregressive image generation, evaluating their in-domain performance and out-of-domain generalization, while scrutinizing the impact of different reward models on their respective capabilities. Our findings reveal that GRPO and DPO exhibit distinct advantages, and crucially, that reward models possessing stronger intrinsic generalization capabilities potentially enhance the generalization potential of the applied RL algorithms. Furthermore, we systematically explore three prevalent scaling strategies to enhance both their in-domain and out-of-domain proficiency, deriving unique insights into efficiently scaling performance for each paradigm. We hope our study paves a new path for inspiring future work on developing more effective RL algorithms to achieve robust CoT reasoning in the realm of autoregressive image generation. Code is released at https://github.com/ZiyuGuo99/Image-Generation-CoT
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。