提出高效扩散变压器框架，提升生成速度与质量。
EDT: An Efficient Diffusion Transformer Framework Inspired by Human-like Sketching
- 受人类素描启发设计轻量架构与注意力调制机制
- 训练与推理速度提升最高达3.93倍，FID更低
- 适合追求高效图像生成的开发者与研究者
基于Transformer的扩散概率模型（DPMs）相较于基于CNN的模型展现出更大潜力，但其高昂的计算成本限制了实际应用。为此，本文提出高效扩散变压器（EDT）框架，包含轻量化模型架构、无需训练的注意力调制矩阵及其受人类素描启发的排列方式。此外，针对EDT设计了一种增强令牌关系学习能力的掩码训练策略。大量实验表明，EDT显著降低了训练与推理开销，在图像生成性能上超越现有Transformer基模型。相比MDTv2对应规模，EDT-S、EDT-B和EDT-XL在训练阶段分别实现3.93x、2.84x、1.92x加速，推理阶段分别实现2.29x、2.29x、2.22x加速，且达到更低的FID值。代码已开源：https://github.com/xinwangChen/EDT。
原文摘要 · Abstract (English)
Transformer-based Diffusion Probabilistic Models (DPMs) have shown more potential than CNN-based DPMs, yet their extensive computational requirements hinder widespread practical applications. To reduce the computation budget of transformer-based DPMs, this work proposes the Efficient Diffusion Transformer (EDT) framework. The framework includes a lightweight-design diffusion model architecture, and a training-free Attention Modulation Matrix and its alternation arrangement in EDT inspired by human-like sketching. Additionally, we propose a token relation-enhanced masking training strategy tailored explicitly for EDT to augment its token relation learning capability. Our extensive experiments demonstrate the efficacy of EDT. The EDT framework reduces training and inference costs and surpasses existing transformer-based diffusion models in image synthesis performance, thereby achieving a significant overall enhancement. With lower FID, EDT-S, EDT-B, and EDT-XL attained speed-ups of 3.93x, 2.84x, and 1.92x respectively in the training phase, and 2.29x, 2.29x, and 2.22x respectively in inference, compared to the corresponding sizes of MDTv2. The source code is released at https://github.com/xinwangChen/EDT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。