用负向视觉查询引导，防止图像风格生成时泄露无关内容。
StyleKeeper: Prevent Content Leakage using Negative Visual Query Guidance
- 通过交换自注意力中的查询，实现负向视觉查询引导。
- 显著降低风格提示中无关内容的意外传递，保持文本指令一致性。
- 适合需要精准风格控制的图像生成场景，如艺术创作与设计。
在文本到图像生成领域，扩散模型已成为强大工具。近期基于图像作为提示的视觉提示方法，实现了对风格和内容更精确的控制。然而，现有方法常出现内容泄露问题，即视觉风格提示中的非预期元素会与目标风格一同传递。为此，本文提出：1）扩展无分类器指导（CFG），采用交换自注意力机制；2）引入负向视觉查询引导（NVQG），通过故意模拟内容泄露场景，将自注意力层中的查询进行替换，从而抑制不必要内容的传递。该方法简单有效，显著减少内容泄露。此外，针对真实图像作为风格提示的情况，我们提供了细致解决方案。在多种风格和文本提示下的广泛评估表明，该方法优于现有方法，能准确反映参考风格并忠实匹配文本提示。代码已公开于GitHub。
原文摘要 · Abstract (English)
In the domain of text-to-image generation, diffusion models have emerged as powerful tools. Recently, studies on visual prompting, where images are used as prompts, have enabled more precise control over style and content. However, existing methods often suffer from content leakage, where undesired elements of the visual style prompt are transferred along with the intended style. To address this issue, we 1) extend classifier-free guidance (CFG) to utilize swapping self-attention and propose 2) negative visual query guidance (NVQG) to reduce the transfer of unwanted contents. NVQG employs negative score by intentionally simulating content leakage scenarios that swap queries instead of key and values of self-attention layers from visual style prompts. This simple yet effective method significantly reduces content leakage. Furthermore, we provide careful solutions for using a real image as visual style prompts. Through extensive evaluation across various styles and text prompts, our method demonstrates superiority over existing approaches, reflecting the style of the references, and ensuring that resulting images match the text prompts. Our code is available \href{https://github.com/naver-ai/StyleKeeper}{here}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。