发现扩散模型中可自动生成分割图的关键层,提升图文生成与理解能力。
Seg4Diff: Unveiling Open-Vocabulary Segmentation in Text-to-Image Diffusion Transformers
- 通过分析跨模态注意力结构,定位文本到图像语义对齐的关键层。
- 该关键层能自然生成高质量分割掩码,且经微调后性能进一步提升。
- 适合关注图文生成、视觉理解与统一模型设计的研究者。
文本到图像扩散模型通过跨模态注意力机制隐式地将文本概念锚定于图像。近期的多模态扩散变压器(MM-DiT)引入图像与文本标记拼接后的联合自注意力机制,实现更丰富的跨模态对齐。然而,对注意力图如何贡献于图像生成的理解仍不充分。本文提出Seg4Diff(用于扩散的分割),系统分析MM-DiT的注意力结构,聚焦特定层级如何将文本语义传播至图像。我们识别出一个语义锚定专家层——特定的MM-DiT模块,始终将文本标记与空间连贯的图像区域对齐,自然产生高质量语义分割掩码。进一步实验表明,使用带有掩码标注的图像数据进行轻量级微调,可增强该层的语义分组能力,从而同时提升分割性能与生成图像保真度。结果表明,语义分组是扩散变压器中的涌现特性,可通过选择性增强来同时优化分割与生成效果,为融合视觉感知与生成的统一模型开辟新路径。
原文摘要 · Abstract (English)
Text-to-image diffusion models excel at translating language prompts into photorealistic images by implicitly grounding textual concepts through their cross-modal attention mechanisms. Recent multi-modal diffusion transformers extend this by introducing joint self-attention over concatenated image and text tokens, enabling richer and more scalable cross-modal alignment. However, a detailed understanding of how and where these attention maps contribute to image generation remains limited. In this paper, we introduce Seg4Diff (Segmentation for Diffusion), a systematic framework for analyzing the attention structures of MM-DiT, with a focus on how specific layers propagate semantic information from text to image. Through comprehensive analysis, we identify a semantic grounding expert layer, a specific MM-DiT block that consistently aligns text tokens with spatially coherent image regions, naturally producing high-quality semantic segmentation masks. We further demonstrate that applying a lightweight fine-tuning scheme with mask-annotated image data enhances the semantic grouping capabilities of these layers and thereby improves both segmentation performance and generated image fidelity. Our findings demonstrate that semantic grouping is an emergent property of diffusion transformers and can be selectively amplified to advance both segmentation and generation performance, paving the way for unified models that bridge visual perception and generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。