通过迭代反馈学习融合多模型优势，提升文本生成图像的组合能力。
IterComp: Iterative Composition-Aware Feedback Learning from Model Gallery for Text-to-Image Generation
- 构建六模型画廊，基于三类组合指标评估模型偏好。
- 提出迭代反馈机制，使模型与奖励模型在多轮中持续优化。
- 显著提升多类别物体组合与复杂语义对齐效果，适合生成复杂场景。
先进扩散模型如 RPG、Stable Diffusion 3 和 FLUX 在组合式文生图任务中取得显著进展，但各自擅长不同方面：有的强于属性绑定，有的擅长空间关系。为整合各模型互补优势，本文提出 IterComp 框架，从多个模型中聚合组合感知偏好，并采用迭代反馈学习提升生成质量。具体地，我们构建由六款强大开源扩散模型组成的画廊，评估其在属性绑定、空间关系和非空间关系三类关键组合指标上的表现，建立包含大量图像-排名对的组合感知模型偏好数据集，用于训练组合感知奖励模型。进一步提出一种闭环迭代反馈学习方法，实现基底扩散模型与奖励模型的渐进式自优化。理论证明其有效性，大量实验表明，相比先前最先进方法（如 Omost 与 FLUX），在多类别物体组合和复杂语义对齐任务上均表现显著更优。IterComp 为扩散模型的奖励反馈学习与组合生成开辟了新方向。代码已公开。
原文摘要 · Abstract (English)
Advanced diffusion models like RPG, Stable Diffusion 3 and FLUX have made notable strides in compositional text-to-image generation. However, these methods typically exhibit distinct strengths for compositional generation, with some excelling in handling attribute binding and others in spatial relationships. This disparity highlights the need for an approach that can leverage the complementary strengths of various models to comprehensively improve the composition capability. To this end, we introduce IterComp, a novel framework that aggregates composition-aware model preferences from multiple models and employs an iterative feedback learning approach to enhance compositional generation. Specifically, we curate a gallery of six powerful open-source diffusion models and evaluate their three key compositional metrics: attribute binding, spatial relationships, and non-spatial relationships. Based on these metrics, we develop a composition-aware model preference dataset comprising numerous image-rank pairs to train composition-aware reward models. Then, we propose an iterative feedback learning method to enhance compositionality in a closed-loop manner, enabling the progressive self-refinement of both the base diffusion model and reward models over multiple iterations. Theoretical proof demonstrates the effectiveness and extensive experiments show our significant superiority over previous SOTA methods (e.g., Omost and FLUX), particularly in multi-category object composition and complex semantic alignment. IterComp opens new research avenues in reward feedback learning for diffusion models and compositional generation. Code: https://github.com/YangLing0818/IterComp
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。