通过时间嵌入优化缓存策略，显著提升视频生成速度。
Timestep Embedding Tells: It's Time to Cache for Video Diffusion Model
- 用时间嵌入预测模型输出差异，指导更优缓存选择
- 实现4.41倍加速，视觉质量损失仅0.07%
- 无需训练，适用于各类扩散模型，适合高效推理场景
作为视频生成的核心架构，扩散模型因去噪过程的顺序性面临推理速度慢的问题。以往方法通过在均匀选取的时间步上缓存和重用模型输出来加速，但忽略了不同时间步间模型输出差异的非均匀性，导致缓存选择不当，难以平衡推理效率与视觉质量。本文提出无需训练的Timestep Embedding Aware Cache（TeaCache）方法，通过估计并利用模型输出在时间步上的波动差异，实现更精准的缓存决策。TeaCache不直接使用计算成本高的模型输出，而是聚焦于与输出强相关的模型输入，并利用时间嵌入对噪声输入进行调制，使其差异更贴近真实输出差异。进一步引入重缩放策略优化估计差异，据此决定是否缓存输出。实验表明，TeaCache在Open-Sora-Plan上实现最高4.41倍加速，视觉质量下降仅0.07%（Vbench分数），效果显著。
原文摘要 · Abstract (English)
As a fundamental backbone for video generation, diffusion models are challenged by low inference speed due to the sequential nature of denoising. Previous methods speed up the models by caching and reusing model outputs at uniformly selected timesteps. However, such a strategy neglects the fact that differences among model outputs are not uniform across timesteps, which hinders selecting the appropriate model outputs to cache, leading to a poor balance between inference efficiency and visual quality. In this study, we introduce Timestep Embedding Aware Cache (TeaCache), a training-free caching approach that estimates and leverages the fluctuating differences among model outputs across timesteps. Rather than directly using the time-consuming model outputs, TeaCache focuses on model inputs, which have a strong correlation with the modeloutputs while incurring negligible computational cost. TeaCache first modulates the noisy inputs using the timestep embeddings to ensure their differences better approximating those of model outputs. TeaCache then introduces a rescaling strategy to refine the estimated differences and utilizes them to indicate output caching. Experiments show that TeaCache achieves up to 4.41x acceleration over Open-Sora-Plan with negligible (-0.07% Vbench score) degradation of visual quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。