用视觉提示生成多样场景,保持物体特征不变。
Object-level Visual Prompts for Compositional Image Generation

- 通过双编码器混合注意力机制,分离布局与细节信息
- 生成图像保留视觉提示的唯一特征,支持灵活组合
- 适合需要精准物体控制的创意生成任务
我们提出一种在文本到图像扩散模型中构建对象级视觉提示的方法。该方法旨在生成跨多样场景和风格的语义连贯组合,类似文本提示的灵活性与表现力。核心挑战在于保持输入视觉提示中物体的身份,同时在不同图像间生成多样化组合。为此,我们引入一种新的KV混合交叉注意力机制,其中键(keys)和值(values)分别来自不同的视觉表征:键由具有小瓶颈的编码器生成以控制布局,值则来自大瓶颈编码器以捕捉精细外观细节。通过融合这两类互补来源的键与值,模型既能保持视觉提示的身份,又能支持物体排列、姿态和构图的灵活变化。推理阶段,我们进一步提出基于对象的组合引导策略,提升身份保留与布局正确性。实验表明,该方法能生成多样且保留各视觉提示独特特征的场景组合,显著拓展了文本到图像生成的创作潜力。
原文摘要 · Abstract (English)
We introduce a method for composing object-level visual prompts within a text-to-image diffusion model. Our approach addresses the task of generating semantically coherent compositions across diverse scenes and styles, similar to the versatility and expressiveness offered by text prompts. A key challenge in this task is to preserve the identity of the objects depicted in the input visual prompts, while also generating diverse compositions across different images. To address this challenge, we introduce a new KV-mixed cross-attention mechanism, in which keys and values are learned from distinct visual representations. The keys are derived from an encoder with a small bottleneck for layout control, whereas the values come from a larger bottleneck encoder that captures fine-grained appearance details. By mixing keys and values from these complementary sources, our model preserves the identity of the visual prompts while supporting flexible variations in object arrangement, pose, and composition. During inference, we further propose object-level compositional guidance to improve the method's identity preservation and layout correctness. Results show that our technique produces diverse scene compositions that preserve the unique characteristics of each visual prompt, expanding the creative potential of text-to-image generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。