通过预测特征缓存加速扩散模型,实现6倍以上提速且画质损失小。
SpeCa: Accelerating Diffusion Transformers with Speculative Feature Caching
- 用推测采样预测后续步骤的中间特征,减少重复计算。
- 在FLUX上提速6.34倍,画质仅降5.5%,视频生成达6.1倍加速。
- 动态分配计算资源,复杂样本重点处理,适合实时生成应用。
扩散模型在高保真图像与视频生成中取得突破,但其计算开销阻碍了实时应用。该模型面临两大挑战:时间依赖性导致无法并行化,以及每步去噪需高成本前向传播。受大语言模型推测解码启发,我们提出SpeCa——一种‘预测-验证’加速框架,有效解决上述问题。核心创新在于引入推测采样,基于已完全计算的参考时间步预测后续时间步的中间特征。方法采用无参数验证机制,高效评估预测可靠性,实现实时接受或拒绝预测,开销极低。此外,引入样本自适应计算分配,根据生成复杂度动态调节资源:简单样本减少计算,复杂样本保留高强度处理。实验表明,对FLUX实现6.34倍加速,画质仅下降5.5%;对DiT提速7.3倍,保持生成保真度;对HunyuanVideo在6.1倍加速下仍达79.84% VBench得分。验证机制开销仅为完整推理的1.67%-3.5%,建立高效扩散模型推理新范式,即使在激进加速比下仍保持高质量生成。代码已开源:https://github.com/Shenyi-Z/Cache4Diffusion。
原文摘要 · Abstract (English)
Diffusion models have revolutionized high-fidelity image and video synthesis, yet their computational demands remain prohibitive for real-time applications. These models face two fundamental challenges: strict temporal dependencies preventing parallelization, and computationally intensive forward passes required at each denoising step. Drawing inspiration from speculative decoding in large language models, we present SpeCa, a novel 'Forecast-then-verify' acceleration framework that effectively addresses both limitations. SpeCa's core innovation lies in introducing Speculative Sampling to diffusion models, predicting intermediate features for subsequent timesteps based on fully computed reference timesteps. Our approach implements a parameter-free verification mechanism that efficiently evaluates prediction reliability, enabling real-time decisions to accept or reject each prediction while incurring negligible computational overhead. Furthermore, SpeCa introduces sample-adaptive computation allocation that dynamically modulates resources based on generation complexity, allocating reduced computation for simpler samples while preserving intensive processing for complex instances. Experiments demonstrate 6.34x acceleration on FLUX with minimal quality degradation (5.5% drop), 7.3x speedup on DiT while preserving generation fidelity, and 79.84% VBench score at 6.1x acceleration for HunyuanVideo. The verification mechanism incurs minimal overhead (1.67%-3.5% of full inference costs), establishing a new paradigm for efficient diffusion model inference while maintaining generation quality even at aggressive acceleration ratios. Our codes have been released in Github: \textbf{https://github.com/Shenyi-Z/Cache4Diffusion}
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。