无需训练即可提升文本生成图像的语义对齐能力
Towards Transformer-Based Aligned Generation with Self-Coherence Guidance
- 通过动态优化交叉注意力图增强生成对齐
- 在复杂文本提示下显著优于现有方法
- 适合需要精准图文对齐的生成任务
我们提出一种全新的、无需训练的方法，用于提升基于Transformer的文本引导扩散模型（TGDMs）的生成对齐性。现有TGDMs在处理复杂文本提示或多概念属性绑定时常出现语义错位。以往针对U-Net架构的方法主要优化潜在空间，但难以直接应用于Transformer结构。本文提出自一致性引导（Self-Coherence Guidance），在生成过程中动态优化交叉注意力图，利用前一去噪步骤的掩码信息进行修正，实现精确对齐且无需额外训练。为验证效果，我们构建了更具挑战性的评估基准，涵盖粗粒度属性绑定、细粒度属性绑定和风格绑定任务。实验结果表明，本方法在所有任务中均显著超越现有最先进方法。代码已开源：https://scg-diffusion.github.io/scg-diffusion。
原文摘要 · Abstract (English)
We introduce a novel, training-free approach for enhancing alignment in Transformer-based Text-Guided Diffusion Models (TGDMs). Existing TGDMs often struggle to generate semantically aligned images, particularly when dealing with complex text prompts or multi-concept attribute binding challenges. Previous U-Net-based methods primarily optimized the latent space, but their direct application to Transformer-based architectures has shown limited effectiveness. Our method addresses these challenges by directly optimizing cross-attention maps during the generation process. Specifically, we introduce Self-Coherence Guidance, a method that dynamically refines attention maps using masks derived from previous denoising steps, ensuring precise alignment without additional training. To validate our approach, we constructed more challenging benchmarks for evaluating coarse-grained attribute binding, fine-grained attribute binding, and style binding. Experimental results demonstrate the superior performance of our method, significantly surpassing other state-of-the-art methods across all evaluated tasks. Our code is available at https://scg-diffusion.github.io/scg-diffusion.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。