让大模型精准控制多个图像元素的属性,避免混淆。
DreamRenderer: Taming Multi-Instance Attribute Control in Large-Scale Text-to-Image Models
- 用复制的图像标记作为桥梁,绑定文本与对应图像区域。
- 只在关键层启用强绑定,提升控制精度17.7%以上。
- 适合需要精细多对象编辑的设计师和研究人员。
基于深度图和Canny边缘图的图像条件生成方法在精确图像合成方面表现出色,但现有模型仍难以准确控制多个实例(或区域)的内容。即使最先进的模型如FLUX和3DIS也存在属性泄露问题,限制了用户控制能力。为此,我们提出DreamRenderer,一种基于FLUX的免训练方法,支持通过边界框或掩码控制每个实例内容,同时保持整体视觉和谐。提出两项关键创新:1)桥接图像标记用于硬性绑定文本属性,利用复制的图像标记作为桥梁,确保仅在文本数据上预训练的T5嵌入能正确绑定每个实例的视觉属性;2)仅在关键层应用硬性图像属性绑定,通过分析FLUX识别出负责实例属性渲染的关键层,仅在此类层使用硬绑定,其余层采用软绑定,以兼顾精度与画质。在COCO-POS和COCO-MIG基准上的评估显示,DreamRenderer相比FLUX将图像成功率提升17.7%,并使布局到图像模型如GLIGEN和3DIS的性能最高提升26.8%。
原文摘要 · Abstract (English)
Image-conditioned generation methods, such as depth- and canny-conditioned approaches, have demonstrated remarkable abilities for precise image synthesis. However, existing models still struggle to accurately control the content of multiple instances (or regions). Even state-of-the-art models like FLUX and 3DIS face challenges, such as attribute leakage between instances, which limits user control. To address these issues, we introduce DreamRenderer, a training-free approach built upon the FLUX model. DreamRenderer enables users to control the content of each instance via bounding boxes or masks, while ensuring overall visual harmony. We propose two key innovations: 1) Bridge Image Tokens for Hard Text Attribute Binding, which uses replicated image tokens as bridge tokens to ensure that T5 text embeddings, pre-trained solely on text data, bind the correct visual attributes for each instance during Joint Attention; 2) Hard Image Attribute Binding applied only to vital layers. Through our analysis of FLUX, we identify the critical layers responsible for instance attribute rendering and apply Hard Image Attribute Binding only in these layers, using soft binding in the others. This approach ensures precise control while preserving image quality. Evaluations on the COCO-POS and COCO-MIG benchmarks demonstrate that DreamRenderer improves the Image Success Ratio by 17.7% over FLUX and enhances the performance of layout-to-image models like GLIGEN and 3DIS by up to 26.8%. Project Page: https://limuloo.github.io/DreamRenderer/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。