通过块级缓存加速视频扩散模型推理,不降质且提速六倍。
BWCache: Accelerating Video Diffusion Transformers through Block-Wise Caching
- 按模块缓存并复用扩散过程中的中间特征
- 在中间步骤特征相似时触发复用,减少冗余计算
- 无需训练,适配多种视频生成模型
最近的扩散变换器(DiT)在视频生成中达到顶尖水平,但其固有的逐步去噪过程带来显著延迟,限制了实际应用。现有加速方法或因结构修改牺牲画质,或未能以恰当粒度复用中间特征。我们分析发现,DiT块是推理延迟的主要来源。在扩散步数中,块特征变化呈现先降后升的U形模式,中间阶段特征高度相似,存在大量计算冗余。本文提出无需训练的块级缓存(BWCache)方法,动态缓存并跨扩散步复用DiT块的特征。同时引入相似性指标,在相邻步特征差异低于阈值时才触发复用,从而最小化冗余计算并保持视觉质量。在多个视频扩散模型上的实验表明,BWCache可实现最高6倍加速,且视觉质量相当。
原文摘要 · Abstract (English)
Recent advancements in Diffusion Transformers (DiTs) have established them as the state-of-the-art method for video generation. However, their inherently sequential denoising process results in inevitable latency, limiting real-world applicability. Existing acceleration methods either compromise visual quality due to architectural modifications or fail to reuse intermediate features at proper granularity. Our analysis reveals that DiT blocks are the primary contributors to inference latency. Across diffusion timesteps, the feature variations of DiT blocks exhibit a U-shaped pattern with high similarity during intermediate timesteps, which suggests substantial computational redundancy. In this paper, we propose Block-Wise Caching (BWCache), a training-free method to accelerate DiT-based video generation. BWCache dynamically caches and reuses features from DiT blocks across diffusion timesteps. Furthermore, we introduce a similarity indicator that triggers feature reuse only when the differences between block features at adjacent timesteps fall below a threshold, thereby minimizing redundant computations while maintaining visual fidelity. Extensive experiments on several video diffusion models demonstrate that BWCache achieves up to 6$\times$ speedup with comparable visual quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。