提出Magnet方法，解决文生图模型属性绑定错误问题
Magnet: We Never Know How Text-to-Image Diffusion Models Work, Until We Learn How Vision-Language Models Function
- 用正负绑定向量分离文本特征，避免属性混淆
- 在Stable Diffusion上提升生成准确率，计算开销极低
- 适合需要精确控制生成内容的创意设计场景
文生图扩散模型（如Stable Diffusion）虽已革新计算机视觉领域，但在处理包含多个属性和物体的复杂提示时，生成质量常下降。已有研究指出，文本嵌入混合是导致属性绑定错误的原因，但缺乏深入探讨。本文深入分析了CLIP文本编码器在理解属性方面的局限性及其对扩散模型的影响，发现文本空间存在属性偏差，并揭示填充嵌入引发概念纠缠的上下文问题。为此，我们提出无需训练的Magnet方法，引入正负绑定向量增强属性解耦，并结合邻近策略提高准确性。大量实验表明，Magnet显著提升生成质量和绑定准确率，且计算成本可忽略不计，使模型能生成非传统和非自然的概念。
原文摘要 · Abstract (English)
Text-to-image diffusion models particularly Stable Diffusion, have revolutionized the field of computer vision. However, the synthesis quality often deteriorates when asked to generate images that faithfully represent complex prompts involving multiple attributes and objects. While previous studies suggest that blended text embeddings lead to improper attribute binding, few have explored this in depth. In this work, we critically examine the limitations of the CLIP text encoder in understanding attributes and investigate how this affects diffusion models. We discern a phenomenon of attribute bias in the text space and highlight a contextual issue in padding embeddings that entangle different concepts. We propose \textbf{Magnet}, a novel training-free approach to tackle the attribute binding problem. We introduce positive and negative binding vectors to enhance disentanglement, further with a neighbor strategy to increase accuracy. Extensive experiments show that Magnet significantly improves synthesis quality and binding accuracy with negligible computational cost, enabling the generation of unconventional and unnatural concepts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。