动态调整计算,让扩散模型生成图像更快更省资源。
DyDiT++: Diffusion Transformers with Timestep and Spatial Dynamics for Efficient Visual Generation
- 根据时间步和空间位置动态调节计算量,避免冗余运算。
- 比基线减少51%计算量,速度提升1.73倍,图像质量接近顶尖水平。
- 支持文生图、视频生成,还可用轻量微调降低使用门槛。
扩散变换器(DiT)在视觉生成中表现优异,但计算开销巨大。我们发现其根源在于静态推理范式,导致某些时间步和空间区域存在冗余计算。为此提出动态扩散变换器(DyDiT),在时间与空间维度上动态调节计算。在此基础上,推出升级版DyDiT++,在三方面优化:一、将生成机制扩展至流匹配,提升通用性;二、拓展至视频生成与文生图等复杂任务,增强应用范围;三、提出基于时间步的动态低秩适配(TD-LoRA),实现参数高效微调。在DiT、SiT、Latte和FLUX等多个模型上实验验证,仅需<3%额外微调迭代,即实现DiT-XL降低51%浮点运算量,硬件速度提升1.73倍,且在ImageNet上保持2.07的竞争力FID分数。代码已开源。
原文摘要 · Abstract (English)
Diffusion Transformer (DiT), an emerging diffusion model for visual generation, has demonstrated superior performance but suffers from substantial computational costs. Our investigations reveal that these costs primarily stem from the static inference paradigm, which inevitably introduces redundant computation in certain diffusion timesteps and spatial regions. To overcome this inefficiency, we propose Dynamic Diffusion Transformer (DyDiT), an architecture that dynamically adjusts its computation along both timestep and spatial dimensions. Building on these designs, we present an extended version, DyDiT++, with improvements in three key aspects. First, it extends the generation mechanism of DyDiT beyond diffusion to flow matching, demonstrating that our method can also accelerate flow-matching-based generation, enhancing its versatility. Furthermore, we enhance DyDiT to tackle more complex visual generation tasks, including video generation and text-to-image generation, thereby broadening its real-world applications. Finally, to address the high cost of full fine-tuning and democratize technology access, we investigate the feasibility of training DyDiT in a parameter-efficient manner and introduce timestep-based dynamic LoRA (TD-LoRA). Extensive experiments on diverse visual generation models, including DiT, SiT, Latte, and FLUX, demonstrate the effectiveness of DyDiT++. Remarkably, with <3% additional fine-tuning iterations, our approach reduces the FLOPs of DiT-XL by 51%, yielding 1.73x realistic speedup on hardware, and achieves a competitive FID score of 2.07 on ImageNet. The code is available at https://github.com/alibaba-damo-academy/DyDiT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。