通过复用相似时空特征,让扩散Transformer生成更快。
BlockDance: Reuse Structurally Similar Spatio-Temporal Features to Accelerate Diffusion Transformers
- 识别相邻时间步中结构相似的特征,优先复用以减少重复计算。
- 在多种模型上实现25%至50%加速,保持生成质量不变。
- 适配不同生成内容,动态分配资源,适合实际部署场景。
扩散模型在图像生成方面表现出色,尤其是采用Transformer架构后显著提升了视觉与艺术质量。然而,扩散Transformer(DiTs)仍面临推理速度慢的问题,主要源于迭代去噪过程。为此,我们提出BlockDance,一种无需训练的加速方法,通过挖掘相邻时间步间的特征相似性来提升效率。不同于以往缺乏多尺度针对性的特征复用策略,BlockDance聚焦于识别结构上最相似的时空特征(即结构相似时空特征,STSS),这些特征主要出现在去噪后期的结构关注块中。通过缓存并重用这些高度相似的特征,有效减少冗余计算,从而加速模型并保持与原始结果的一致性。针对生成内容多样性和冗余特征分布差异,我们进一步引入BlockDance-Ada——一个轻量级决策网络,实现实例级资源动态分配,保障生成质量。两者在多个生成任务和模型上均表现优异,加速比达25%~50%,且生成质量无损。
原文摘要 · Abstract (English)
Diffusion models have demonstrated impressive generation capabilities, particularly with recent advancements leveraging transformer architectures to improve both visual and artistic quality. However, Diffusion Transformers (DiTs) continue to encounter challenges related to low inference speed, primarily due to the iterative denoising process. To address this issue, we propose BlockDance, a training-free approach that explores feature similarities at adjacent time steps to accelerate DiTs. Unlike previous feature-reuse methods that lack tailored reuse strategies for features at different scales, BlockDance prioritizes the identification of the most structurally similar features, referred to as Structurally Similar Spatio-Temporal (STSS) features. These features are primarily located within the structure-focused blocks of the transformer during the later stages of denoising. BlockDance caches and reuses these highly similar features to mitigate redundant computation, thereby accelerating DiTs while maximizing consistency with the generated results of the original model. Furthermore, considering the diversity of generated content and the varying distributions of redundant features, we introduce BlockDance-Ada, a lightweight decision-making network tailored for instance-specific acceleration. BlockDance-Ada dynamically allocates resources and provides superior content quality. Both BlockDance and BlockDance-Ada have proven effective across various generation tasks and models, achieving accelerations between 25% and 50% while maintaining generation quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。