3DIS通过深度图分离布局与属性生成,实现文本到图像的精准多实例合成。
3DIS: Depth-Driven Decoupled Instance Synthesis for Text-to-Image Generation
- 分两阶段生成:先用深度图定位实例,再用ControlNet渲染细节属性
- 在COCO-Position和COCO-MIG上布局精度与属性还原度显著优于现有方法
- 无需微调即可适配SD2、SDXL等主流模型,适合需要精确控制的生成场景
文本到图像生成对可控输出的需求日益增长,推动了多实例生成(MIG)的发展,使用户可定义实例布局与属性。然而,与ControlNet等图像条件生成方法相比,现有MIG技术尚未广泛集成至SD2和SDXL等先进模型,主要受限于难以构建同时处理实例定位与属性渲染的鲁棒渲染器。本文提出深度驱动的解耦实例合成框架3DIS,将MIG过程分为两个阶段:(i) 生成粗略场景深度图以实现精确的实例定位与场景构图;(ii) 利用预训练ControlNet在任意基础模型上无须额外训练即可渲染细粒度属性。3DIS通过定制适配器融入LDM3D实现基于深度的精准布局,并采用免微调方法提升实例级属性渲染效果。在COCO-Position与COCO-MIG基准上的大量实验表明,3DIS在布局精度和属性渲染方面均显著超越现有方法。尤为关键的是,3DIS与多种基础模型无缝兼容,为高级多实例生成提供了一种强大且灵活的解决方案。代码已开源:https://github.com/limuloo/3DIS。
原文摘要 · Abstract (English)
The increasing demand for controllable outputs in text-to-image generation has spurred advancements in multi-instance generation (MIG), allowing users to define both instance layouts and attributes. However, unlike image-conditional generation methods such as ControlNet, MIG techniques have not been widely adopted in state-of-the-art models like SD2 and SDXL, primarily due to the challenge of building robust renderers that simultaneously handle instance positioning and attribute rendering. In this paper, we introduce Depth-Driven Decoupled Instance Synthesis (3DIS), a novel framework that decouples the MIG process into two stages: (i) generating a coarse scene depth map for accurate instance positioning and scene composition, and (ii) rendering fine-grained attributes using pre-trained ControlNet on any foundational model, without additional training. Our 3DIS framework integrates a custom adapter into LDM3D for precise depth-based layouts and employs a finetuning-free method for enhanced instance-level attribute rendering. Extensive experiments on COCO-Position and COCO-MIG benchmarks demonstrate that 3DIS significantly outperforms existing methods in both layout precision and attribute rendering. Notably, 3DIS offers seamless compatibility with diverse foundational models, providing a robust, adaptable solution for advanced multi-instance generation. The code is available at: https://github.com/limuloo/3DIS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。