提出适配多模态扩散Transformer的双向编辑方法
Exploring Multimodal Diffusion Transformers for Enhanced Prompt-based Image Editing
- 设计统一注意力机制实现文本与图像双向信息流动
- 通过四块分解揭示注意力矩阵内在特性，支持跨变体编辑
- 适用于少步模型，为新架构提供可迁移编辑思路
基于Transformer的扩散模型已取代传统U-Net架构，多模态扩散Transformer（MM-DiT）成为Stable Diffusion 3和Flux等先进模型的核心。以往方法依赖单向交叉注意力，信息仅从文本嵌入流向图像隐变量。相比之下，MM-DiT采用统一注意力机制，将双模态输入投影拼接后进行单一全注意力操作，实现文本与图像分支间的双向信息流动。这一架构变革给现有编辑技术带来挑战。本文系统分析MM-DiT的注意力机制，通过将注意力矩阵分解为四个独立区块，揭示其内在特征。基于此，提出一种稳健的提示驱动图像编辑方法，可支持多种MM-DiT变体中的全局至局部编辑，包括少步模型。研究成果弥合了现有U-Net方法与新兴架构之间的差距，深化了对MM-DiT行为模式的理解。
原文摘要 · Abstract (English)
Transformer-based diffusion models have recently superseded traditional U-Net architectures, with multimodal diffusion transformers (MM-DiT) emerging as the dominant approach in state-of-the-art models like Stable Diffusion 3 and Flux.1. Previous approaches have relied on unidirectional cross-attention mechanisms, with information flowing from text embeddings to image latents. In contrast, MMDiT introduces a unified attention mechanism that concatenates input projections from both modalities and performs a single full attention operation, allowing bidirectional information flow between text and image branches. This architectural shift presents significant challenges for existing editing techniques. In this paper, we systematically analyze MM-DiT's attention mechanism by decomposing attention matrices into four distinct blocks, revealing their inherent characteristics. Through these analyses, we propose a robust, prompt-based image editing method for MM-DiT that supports global to local edits across various MM-DiT variants, including few-step models. We believe our findings bridge the gap between existing U-Net-based methods and emerging architectures, offering deeper insights into MMDiT's behavioral patterns.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。