针对扩散模型中多实例生成难题,提出分层渐进注意力调优方法。
Hierarchical and Step-Layer-Wise Tuning of Attention Specialty for Multi-Instance Synthesis in Diffusion Transformers
- 基于分层注意力分析,发现实例、背景、属性令牌在不同层主导响应。
- 无需训练,通过分层渐进调优提升复杂布局生成精度,定位更准。
- 适合需要精准控制多物体布局的图像生成应用,如设计与可视化。
文本到图像生成模型在多实例合成(MIS)任务中常面临挑战,需根据复杂提示准确描绘图像中多个独立实体。传统UNet架构(如SD v1.5/SDXL)的MIS控制方法无法适配依赖图像-文本令牌集成注意力的DiT架构(如FLUX和SD v3.5)。本文首先分析DiT中的混合注意力机制,通过逐标记与逐层注意力图分析,揭示其分层响应结构:早期层由实例令牌主导,中期层为背景令牌,后期层则由属性令牌主导。基于此,提出一种无需训练的分层渐进注意力特化调优(AST)方法,通过在不同层与生成步骤中放大关键区域、抑制无关区域,优化多模态交互。该方法通过实例级草图实现复杂提示的分层解耦。在升级版草图布局的T2I-CompBench及定制复杂场景上评估,定量与定性结果均表明,本方法显著提升复杂布局生成能力,确保多实例位置与属性表征更精确。
原文摘要 · Abstract (English)
Text-to-image (T2I) generation models often struggle with multi-instance synthesis (MIS), where they must accurately depict multiple distinct instances in a single image based on complex prompts detailing individual features. Traditional MIS control methods for UNet architectures like SD v1.5/SDXL fail to adapt to DiT-based models like FLUX and SD v3.5, which rely on integrated attention between image and text tokens rather than text-image cross-attention. To enhance MIS in DiT, we first analyze the mixed attention mechanism in DiT. Our token-wise and layer-wise analysis of attention maps reveals a hierarchical response structure: instance tokens dominate early layers, background tokens in middle layers, and attribute tokens in later layers. Building on this observation, we propose a training-free approach for enhancing MIS in DiT-based models with hierarchical and step-layer-wise attention specialty tuning (AST). AST amplifies key regions while suppressing irrelevant areas in distinct attention maps across layers and steps, guided by the hierarchical structure. This optimizes multimodal interactions by hierarchically decoupling the complex prompts with instance-based sketches. We evaluate our approach using upgraded sketch-based layouts for the T2I-CompBench and customized complex scenes. Both quantitative and qualitative results confirm our method enhances complex layout generation, ensuring precise instance placement and attribute representation in MIS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。