解决相似物体生成时的混淆问题，提升图像合成准确性。
Self-Cross Diffusion Guidance for Text-to-Image Synthesis of Similar Subjects
- 通过自交叉注意力惩罚减少相似物体间的混淆。
- 在多个相似物体场景中显著降低主体混合错误率。
- 无需训练，适配主流扩散模型，适合图像生成研究者。
扩散模型在图像、视频、3D资产生成方面取得了前所未有的保真度与多样性。然而，对于生成多个外观相似的主体时，主体混淆仍是未解难题。本文提出自交叉注意力引导机制，通过惩罚跨注意力图与聚合自注意力图之间的重叠来缓解该问题。相比仅依赖自注意力或跨注意力的方法，本方法能更有效消除主体混淆，并作用于所有相关区域（如鸟喙等非显著部位）。对每个主体，我们聚合跨注意力值较高的局部区域的自注意力图，形成其整体关注区域。该无训练方法可提升基于U-Net和Transformer的扩散模型性能，如Stable Diffusion系列。我们还发布了相似主体数据集（SSD），并使用GPT-4o进行自动可靠评估。大量定性与定量结果验证了该方法的有效性。
原文摘要 · Abstract (English)
Diffusion models achieved unprecedented fidelity and diversity for synthesizing image, video, 3D assets, etc. However, subject mixing is an unresolved issue for diffusion-based image synthesis, particularly for synthesizing multiple similar-looking subjects. We propose Self-Cross Diffusion Guidance to penalize the overlap between cross-attention maps and the aggregated self-attention map. Compared to previous methods based on self-attention or cross-attention alone, our guidance is more effective in eliminating subject mixing. What's more, our guidance addresses subject mixing for all relevant patches beyond the most discriminant one, e.g., the beak of a bird. For each subject, we aggregate self-attention maps of patches with higher cross-attention values. Thus, the aggregated self-attention map forms a region that the whole subject attends to. Our training-free method boosts the performance of both Unet-based and Transformer-based diffusion models such as the Stable Diffusion series. We also release a similar subjects dataset (SSD), a challenging benchmark, and utilize GPT-4o for automatic and reliable evaluation. Extensive qualitative and quantitative results demonstrate the effectiveness of our self-cross diffusion guidance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。