通过复用扩散模型相邻步骤的相似特征，实现跨模态加速推理。
SmoothCache: A Universal Inference Acceleration Technique for Diffusion Transformers
- 利用相邻时间步层输出高度相似性，自适应缓存关键特征。
- 在图像、视频、音频生成上提速8%至71%，质量不降反升。
- 无需修改模型，适配DiT-XL、Open-Sora等主流生成模型。
扩散Transformer（DiT）已成为图像、视频和语音合成等多种任务的强大生成模型。然而，其推理过程因反复计算资源密集的注意力与前馈模块而十分耗时。为此，我们提出SmoothCache，一种适用于DiT架构的通用模型无关推理加速技术。SmoothCache利用相邻扩散时间步间层输出的高度相似性，通过小规模校准集分析逐层表示误差，自适应地缓存并重用关键特征。实验表明，SmoothCache在多种模态下实现8%至71%的加速，同时保持甚至提升生成质量。我们在DiT-XL图像生成、Open-Sora文本到视频生成及Stable Audio Open文本到音频生成上验证了其有效性，展现了其实现实时应用与降低高性能DiT模型使用门槛的巨大潜力。
原文摘要 · Abstract (English)
Diffusion Transformers (DiT) have emerged as powerful generative models for various tasks, including image, video, and speech synthesis. However, their inference process remains computationally expensive due to the repeated evaluation of resource-intensive attention and feed-forward modules. To address this, we introduce SmoothCache, a model-agnostic inference acceleration technique for DiT architectures. SmoothCache leverages the observed high similarity between layer outputs across adjacent diffusion timesteps. By analyzing layer-wise representation errors from a small calibration set, SmoothCache adaptively caches and reuses key features during inference. Our experiments demonstrate that SmoothCache achieves 8% to 71% speed up while maintaining or even improving generation quality across diverse modalities. We showcase its effectiveness on DiT-XL for image generation, Open-Sora for text-to-video, and Stable Audio Open for text-to-audio, highlighting its potential to enable real-time applications and broaden the accessibility of powerful DiT models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。