用实例组装注意力提升布局控制,生成更精准的图像。
InstanceAssemble: Layout-Aware Image Generation via Instance Assembling Attention
- 通过实例组装注意力融合布局框与多模态内容,实现精细位置控制。
- 在90k实例的Denselayout基准上达到当前最优性能。
- 兼容多种风格LoRA模块,适合需要高可控性的图像生成任务。
扩散模型在生成高质量图像方面表现出色。近期的布局到图像(L2I)生成方法利用位置条件和文本描述,实现了精确可控的图像合成。尽管整体进展显著,现有L2I方法仍存在性能不足的问题。为此,我们提出InstanceAssemble,一种通过实例组装注意力引入布局条件的新架构,支持以边界框(bbox)进行位置控制,并可同时处理文本及额外视觉内容的多模态内容控制。该方法通过轻量级LoRA模块,灵活适配现有的基于DiT的文本到图像(T2I)模型。此外,我们构建了全新的布局到图像基准Denselayout,包含5,000张图像、共计90,000个实例。我们还提出了布局定位得分(LGS),一种可解释的评估指标,用于更准确地衡量L2I生成的准确性。实验表明,我们的InstanceAssemble方法在复杂布局条件下达到最先进水平,且对多种风格LoRA模块具有强兼容性。代码与预训练模型已公开于https://github.com/FireRedTeam/InstanceAssemble。
原文摘要 · Abstract (English)
Diffusion models have demonstrated remarkable capabilities in generating high-quality images. Recent advancements in Layout-to-Image (L2I) generation have leveraged positional conditions and textual descriptions to facilitate precise and controllable image synthesis. Despite overall progress, current L2I methods still exhibit suboptimal performance. Therefore, we propose InstanceAssemble, a novel architecture that incorporates layout conditions via instance-assembling attention, enabling position control with bounding boxes (bbox) and multimodal content control including texts and additional visual content. Our method achieves flexible adaption to existing DiT-based T2I models through light-weighted LoRA modules. Additionally, we propose a Layout-to-Image benchmark, Denselayout, a comprehensive benchmark for layout-to-image generation, containing 5k images with 90k instances in total. We further introduce Layout Grounding Score (LGS), an interpretable evaluation metric to more precisely assess the accuracy of L2I generation. Experiments demonstrate that our InstanceAssemble method achieves state-of-the-art performance under complex layout conditions, while exhibiting strong compatibility with diverse style LoRA modules. The code and pretrained models are publicly available at https://github.com/FireRedTeam/InstanceAssemble.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。