动态调整计算量，让扩散模型生成更快更省力
Dynamic Diffusion Transformer
- 按生成阶段和空间位置动态调节计算规模
- 减少51%算力消耗，速度提升1.73倍，保持高画质
- 适合追求高效图像生成的研究者与开发者
扩散变换器（DiT）在图像生成中表现优异，但计算成本高昂。研究发现，这源于静态推理范式导致特定扩散步骤和空间区域存在冗余计算。为此，我们提出动态扩散变换器（DyDiT），在生成过程中沿时间步和空间维度动态调整计算。具体地，引入时间步动态宽度（TDW）机制，根据生成阶段自适应调整模型宽度；设计空间动态令牌（SDT）策略，避免在非必要位置进行冗余计算。在多个数据集和不同规模模型上的实验验证了其优越性。仅需不到3%的额外微调迭代，即可将DiT-XL的FLOPs降低51%，生成速度提升1.73倍，在ImageNet上取得2.07的竞争力FID分数。代码已开源。
原文摘要 · Abstract (English)
Diffusion Transformer (DiT), an emerging diffusion model for image generation, has demonstrated superior performance but suffers from substantial computational costs. Our investigations reveal that these costs stem from the static inference paradigm, which inevitably introduces redundant computation in certain diffusion timesteps and spatial regions. To address this inefficiency, we propose Dynamic Diffusion Transformer (DyDiT), an architecture that dynamically adjusts its computation along both timestep and spatial dimensions during generation. Specifically, we introduce a Timestep-wise Dynamic Width (TDW) approach that adapts model width conditioned on the generation timesteps. In addition, we design a Spatial-wise Dynamic Token (SDT) strategy to avoid redundant computation at unnecessary spatial locations. Extensive experiments on various datasets and different-sized models verify the superiority of DyDiT. Notably, with <3% additional fine-tuning iterations, our method reduces the FLOPs of DiT-XL by 51%, accelerates generation by 1.73, and achieves a competitive FID score of 2.07 on ImageNet. The code is publicly available at https://github.com/NUS-HPC-AI-Lab/ Dynamic-Diffusion-Transformer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。