用FLUX模型提升多实例图像生成精度,无需重训适配器。
3DIS-FLUX: simple and efficient multi-instance generation with DiT rendering
- 分两阶段生成:先构图后渲染,仅在构图阶段需训练适配器。
- 基于注意力掩码调控实现每个实例的精细属性控制,图像质量更优。
- 适合需要高效、高质可控图像生成的研究者与开发者使用。
文本到图像生成中对可控输出的需求推动了多实例生成(MIG)的发展,使用户能够定义实例布局与属性。当前主流方法依赖适配器,但每当新模型发布时都需重新训练,资源消耗大。为此提出深度驱动的解耦实例合成(3DIS)框架,将MIG分为深度引导场景构建与预训练深度控制模型的细节渲染两阶段。3DIS仅在场景构建阶段需适配器训练,即可实现免训练的细节渲染。此前3DIS仅使用U-Net架构如SD1.5、SD2和SDXL,未探索最近的DiT架构如FLUX。本文提出3DIS-FLUX,将FLUX.1-Depth-dev模型用于深度图控制生成,并设计细节渲染器,通过布局信息调控FLUX的联合注意力机制中的注意力掩码,实现每个实例的细粒度属性精确渲染。实验表明,3DIS-FLUX在性能和图像质量上均优于原3DIS(使用SD2/SDXL),并超越当前最先进适配器方法。
原文摘要 · Abstract (English)
The growing demand for controllable outputs in text-to-image generation has driven significant advancements in multi-instance generation (MIG), enabling users to define both instance layouts and attributes. Currently, the state-of-the-art methods in MIG are primarily adapter-based. However, these methods necessitate retraining a new adapter each time a more advanced model is released, resulting in significant resource consumption. A methodology named Depth-Driven Decoupled Instance Synthesis (3DIS) has been introduced, which decouples MIG into two distinct phases: 1) depth-based scene construction and 2) detail rendering with widely pre-trained depth control models. The 3DIS method requires adapter training solely during the scene construction phase, while enabling various models to perform training-free detail rendering. Initially, 3DIS focused on rendering techniques utilizing U-Net architectures such as SD1.5, SD2, and SDXL, without exploring the potential of recent DiT-based models like FLUX. In this paper, we present 3DIS-FLUX, an extension of the 3DIS framework that integrates the FLUX model for enhanced rendering capabilities. Specifically, we employ the FLUX.1-Depth-dev model for depth map controlled image generation and introduce a detail renderer that manipulates the Attention Mask in FLUX's Joint Attention mechanism based on layout information. This approach allows for the precise rendering of fine-grained attributes of each instance. Our experimental results indicate that 3DIS-FLUX, leveraging the FLUX model, outperforms the original 3DIS method, which utilized SD2 and SDXL, and surpasses current state-of-the-art adapter-based methods in terms of both performance and image quality. Project Page: https://limuloo.github.io/3DIS/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。