让参考图中的局部特征精准控制生成图像,突破全局风格的限制。
RefDiT: Local Attribute Guidance in Reference-Based Image Generation

- 用属性分解技术拆解标识符,实现局部区域精准引导
- 通过上下文调整训练低秩适配器,提升细节控制力
- 适合需要精细个性化生成的场景,如人物服装定制
个性化模型利用少量主体参考图生成新图像,而风格迁移方法则基于参考图的全局风格进行生成。现有方法在参考图仅含单一物体时表现良好,能有效捕捉包含所有隐含属性的全局风格。然而,在包含多个具有不同属性特征物体的复杂真实场景中,由于依赖全局引导,这些方法无法定位参考图中的相关元素,限制了基于局部属性生成新图像的能力。此外,现有方法通常使用单个标识符令牌捕捉参考图全部细节,缺乏属性级别的独立控制。为此,我们提出RefDiT,一种新的参考引导图像生成框架。RefDiT 输入参考图、文本提示和可选用户提供的引导上下文,通过属性级分解标识符令牌,构建属性感知的条件信号,并在推理提示中对扩散变换器(DiT)基生成模型的低秩适配器(LoRA)块进行上下文调整训练。RefDiT 学习标识符令牌与参考图局部区域之间的对应关系,从而实现更有效的局部引导。
原文摘要 · Abstract (English)
Personalization models generate new images guided by a few subject references, while style transfer methods aim to produce images aligned with a global style derived from a reference image. Recent approaches perform well when the reference image contains a single object, effectively capturing a global style that encompasses all implicit attributes. However, when applied to complex real-world scenes containing multiple objects with distinct attribute characteristics, these methods, due to their global-level guidance, fail to localize relevant elements in the reference image. The global guidance restricts their ability to generate new images based on the local attributes in the reference image. Moreover, existing methods typically employ a single identifier token to capture all details from the reference, resulting in a lack of individual, attribute-level control. Motivated by these limitations, we propose RefDiT, a novel framework for reference-guided image generation. RefDiT takes as input a reference image, a text prompt, and an optional user-provided guidance context. RefDiT employs local region guidance using the attributes of local elements. It constructs an attribute-aware conditioning signal from the reference image by performing attribute-level decomposition of the identifier token and performs context adjustment in the inference prompt to train low-rank adapter (LoRA) blocks of a diffusion transformer (DiT)-based generative model. RefDiT learns the correspondence between identifier tokens and local regions in the reference image, enabling more effective local guidance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。