改进扩散模型的注意力机制，提升效率与质量
PiT: Progressive Diffusion Transformer
- 用伪移位窗口注意力减少全局计算冗余
- 新架构在更低算力下实现54%的FID提升
- 适合关注高效图像生成的研究者
扩散Transformer（DiTs）通过Transformer结构在图像生成中取得显著成果。传统DiTs采用串行堆叠的全连接全局建模结构，面临二次方复杂度问题。我们通过实证分析发现，DiTs对全局信息依赖程度低于预期，多数层存在显著的全局计算冗余。此外，传统注意力机制存在低频惯性，影响效率。为此，我们提出伪移位窗口注意力（PSWA），通过窗口注意力实现适度的全局-局部信息融合，并引入高频桥接分支模拟移位窗口操作，增强高频信息与跨窗口连接。同时提出渐进式通道分配策略（PCCA），在不增加计算成本前提下捕捉高阶注意力。基于上述创新，我们构建了一系列伪渐进扩散变压器（PiT）。大量实验表明其性能优越：例如，所提PiT-L相比DiT-XL/2实现54%的FID降低，且计算量更少。
原文摘要 · Abstract (English)
Diffusion Transformers (DiTs) achieve remarkable performance within image generation via the transformer architecture. Conventionally, DiTs are constructed by stacking serial isotropic global modeling transformers, which face significant quadratic computational cost. However, through empirical analysis, we find that DiTs do not rely as heavily on global information as previously believed. In fact, most layers exhibit significant redundancy in global computation. Additionally, conventional attention mechanisms suffer from low-frequency inertia, limiting their efficiency. To address these issues, we propose Pseudo Shifted Window Attention (PSWA), which fundamentally mitigates global attention redundancy. PSWA achieves moderate global-local information through window attention. It further utilizes a high-frequency bridging branch to simulate shifted window operations, which both enrich the high-frequency information and strengthen inter-window connections. Furthermore, we propose the Progressive Coverage Channel Allocation (PCCA) strategy that captures high-order attention without additional computational cost. Based on these innovations, we propose a series of Pseudo Progressive Diffusion Transformer (PiT). Our extensive experiments show their superior performance; for example, our proposed PiT-L achieves 54% FID improvement over DiT-XL/2 while using less computation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。