让扩散模型像人一样思考，通过反思迭代生成符合逻辑的图像。
Self-Reflective Reinforcement Learning for Diffusion-based Image Reasoning Generation
- 用自我反思强化学习，让扩散模型在生成中不断迭代修正。
- 首次实现遵循物理规律和非常规现象的逻辑图像生成，性能超GPT-4o。
- 适合需要高逻辑性图像生成的研究者与创意设计人员。
扩散模型在图像生成任务中表现出色，但现有方法在逻辑导向的图像生成中仍面临困境。受大语言模型中思维链（CoT）与强化学习（RL）成功的启发，本文提出SRRL——一种面向扩散模型的自反性强化学习算法，通过在生成轨迹中进行反思与迭代，实现逻辑图像的推理生成。去噪过程中的中间样本含有噪声，导致奖励评估困难。SRRL将整个去噪轨迹视为一个思维链步骤，并引入条件引导前向过程，实现思维链步骤间的反射式迭代。基于SRRL的迭代扩散训练，首次在遵循物理定律及非典型物理现象的任务中引入了思维链式的图像推理生成。案例研究显示，该算法性能甚至优于GPT-4o。
原文摘要 · Abstract (English)
Diffusion models have recently demonstrated exceptional performance in image generation task. However, existing image generation methods still significantly suffer from the dilemma of image reasoning, especially in logic-centered image generation tasks. Inspired by the success of Chain of Thought (CoT) and Reinforcement Learning (RL) in LLMs, we propose SRRL, a self-reflective RL algorithm for diffusion models to achieve reasoning generation of logical images by performing reflection and iteration across generation trajectories. The intermediate samples in the denoising process carry noise, making accurate reward evaluation difficult. To address this challenge, SRRL treats the entire denoising trajectory as a CoT step with multi-round reflective denoising process and introduces condition guided forward process, which allows for reflective iteration between CoT steps. Through SRRL-based iterative diffusion training, we introduce image reasoning through CoT into generation tasks adhering to physical laws and unconventional physical phenomena for the first time. Notably, experimental results of case study exhibit that the superior performance of our SRRL algorithm even compared with GPT-4o. The project page is https://jadenpan0.github.io/srrl.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。