根据视频复杂度动态缓存计算，加速扩散模型生成
Adaptive Caching for Faster Video Generation with Diffusion Transformers
- 按视频运动复杂度智能缓存中间结果，减少重复计算
- 在保持画质前提下，最高提速4.7倍（720p, 2秒视频）
- 无需训练，适配多种扩散模型，适合实时视频生成场景
生成时间一致的高保真视频计算成本高昂，尤其在长时序场景下。近年来的扩散变换器（DiTs）虽取得进展，但因模型更大、注意力机制更重，推理速度更慢。本文提出一种无需训练的加速方法——自适应缓存（AdaCache），基于‘并非所有视频都需要同等计算量’这一观察：不同视频所需去噪步骤差异显著。通过在扩散过程中缓存计算，并为每段视频定制缓存策略，实现质量与延迟的最佳权衡。进一步引入运动正则化（MoReg）机制，利用视频内容动态调节计算分配。该方法作为即插即用模块，在多个视频扩散模型基线中实现显著加速（如Open-Sora 720p - 2s视频生成最高达4.7倍），且不牺牲生成质量。
原文摘要 · Abstract (English)
Generating temporally-consistent high-fidelity videos can be computationally expensive, especially over longer temporal spans. More-recent Diffusion Transformers (DiTs) -- despite making significant headway in this context -- have only heightened such challenges as they rely on larger models and heavier attention mechanisms, resulting in slower inference speeds. In this paper, we introduce a training-free method to accelerate video DiTs, termed Adaptive Caching (AdaCache), which is motivated by the fact that "not all videos are created equal": meaning, some videos require fewer denoising steps to attain a reasonable quality than others. Building on this, we not only cache computations through the diffusion process, but also devise a caching schedule tailored to each video generation, maximizing the quality-latency trade-off. We further introduce a Motion Regularization (MoReg) scheme to utilize video information within AdaCache, essentially controlling the compute allocation based on motion content. Altogether, our plug-and-play contributions grant significant inference speedups (e.g. up to 4.7x on Open-Sora 720p - 2s video generation) without sacrificing the generation quality, across multiple video DiT baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。