让图像提示更精准地影响生成结果,解决冲突与平衡难题。
Devil is in the Detail: Towards Injecting Fine Details of Image Prompt in Image Generation via Conflict-free Guidance and Stratified Attention

- 提出无冲突引导机制,仅将图像提示作为期望条件
- 设计分层注意力,融合双源特征提升对齐与真实感
- 在三类任务中显著优于现有方法,细节还原更准确
大规模文本到图像扩散模型虽能生成高质量、多样化的图像,但难以捕捉如纹理等细微细节,导致用户意图无法充分体现。为此,研究转向以用户提供的图像作为提示(图像提示)生成图像。现有方法通过修改自注意力机制,用图像提示的键和值替换或拼接生成图像的键和值,使自注意力层类似交叉注意力,用于引入文本提示。本文发现现有方法存在两个关键问题:其一,在无分类器引导中忽略图像提示的重要性,导致图像提示同时作为期望与非期望条件,引发信号冲突;为此提出无冲突引导,仅将图像提示作为期望条件,确保生成图像忠实反映提示内容。其二,现有自注意力修改方式在真实感与对齐度间存在权衡:从图像提示选取更多键值可提升对齐度,但从生成图像选取更多则增强真实感。为兼顾两者,提出分层注意力,联合使用两图像的键和值,而非单一选择。在三项图像生成任务上的大量实验表明,所提方法在忠实还原图像提示细节方面显著优于现有模型。
原文摘要 · Abstract (English)
While large-scale text-to-image diffusion models enable the generation of high-quality, diverse images from text prompts, these prompts struggle to capture intricate details, such as textures, preventing the user intent from being reflected. This limitation has led to efforts to generate images conditioned on user-provided images, referred to as image prompts. Recent work modifies the self-attention mechanism to impose image conditions in generated images by replacing or concatenating the keys and values from the image prompt. This enables the self-attention layer to work like a cross-attention layer, generally used to incorporate text prompts. In this paper, we identify two common issues in existing methods of modifying self-attention to generate images that reflect the details of image prompts. First, existing approaches neglect the importance of image prompts in classifier-free guidance. Specifically, current methods use image prompts as both desired and undesired conditions in classifier-free guidance, causing conflicting signals. To resolve this, we propose conflict-free guidance by using image prompts only as desired conditions, ensuring that the generated image faithfully reflects the image prompt. In addition, we observe that the two most common self-attention modifications involve a trade-off between the realism of the generated image and alignment with the image prompt. Specifically, selecting more keys and values from the image prompt improves alignment, while selecting more from the generated image enhances realism. To balance both, we propose an new self-attention modification method, Stratified Attention to jointly use keys and values from both images rather than selecting between them. Through extensive experiments across three image generation tasks, we show that the proposed method outperforms existing image-prompting models in faithfully reflecting the image prompt.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。