动态调整图像生成中补丁大小，加速扩散Transformer推理
DDiT: Dynamic Patch Scheduling for Efficient Diffusion Transformers
- 根据内容复杂度和去噪阶段动态调整补丁大小
- 在FLUX-1.Dev上提速3.52倍，视频模型提速3.2倍
- 适合追求高效生成且不牺牲质量的开发者使用
扩散Transformer（DiTs）在图像与视频生成中达到顶尖性能，但计算开销巨大。主要原因是固定的分块策略：在整个去噪过程中始终使用固定大小的补丁，无论内容复杂度如何。本文提出动态分块（dynamic tokenization），一种高效的测试时策略，根据内容复杂度和去噪步数动态调整补丁尺寸。核心洞察是：早期去噪步骤只需粗粒度补丁以建模全局结构，后期则需更细粒度（更小）补丁以细化局部细节。推理时，该方法在图像与视频生成中动态重分配补丁大小，显著降低计算成本，同时保持感知质量。大量实验表明，该方法在FLUX-1.Dev上实现最高3.52倍、在Wan 2.1上实现3.2倍的加速，且不损失生成质量与提示遵循能力。
原文摘要 · Abstract (English)
Diffusion Transformers (DiTs) have achieved state-of-the-art performance in image and video generation, but their success comes at the cost of heavy computation. This inefficiency is largely due to the fixed tokenization process, which uses constant-sized patches throughout the entire denoising phase, regardless of the content's complexity. We propose dynamic tokenization, an efficient test-time strategy that varies patch sizes based on content complexity and the denoising timestep. Our key insight is that early timesteps only require coarser patches to model global structure, while later iterations demand finer (smaller-sized) patches to refine local details. During inference, our method dynamically reallocates patch sizes across denoising steps for image and video generation and substantially reduces cost while preserving perceptual generation quality. Extensive experiments demonstrate the effectiveness of our approach: it achieves up to $3.52\times$ and $3.2\times$ speedup on FLUX-1.Dev and Wan $2.1$, respectively, without compromising the generation quality and prompt adherence.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。