arXiv:2603.24260cs.CVcs.AI2026-03中稿 · CVPR被引 2

通过智能缓存减少冗余计算,让视频生成更快更高效。

Accelerating Diffusion-based Video Editing via Heterogeneous Caching: Beyond Full Computing at Sampled Denoising Timestep

  • 按上下文相关性区分关键与非关键帧,只缓存重要信息。
  • 在不损失编辑质量前提下,速度提升2.67倍,降低算力消耗。
  • 无需训练,适配主流扩散模型,适合追求效率的开发者。

基于扩散模型的视频编辑已成为高质量、灵活内容生成的重要范式。然而,尽管具备通用性和强大建模能力,扩散变压器(DiT)因迭代去噪过程仍计算成本高昂,限制了实际部署。现有加速方法主要依赖去噪步骤级特征复用,虽缓解了过程冗余,却忽略了DiT架构内部的冗余问题——许多时空标记的注意力操作重复执行,对输出贡献甚微。本文提出HetCache,一种无需训练的扩散加速框架,利用掩码视频到视频(MV2V)生成与编辑中的固有异质性。不同于统一复用或随机采样,HetCache评估指定计算步骤中各类标记的上下文相关性与交互强度,结合空间先验将时空标记划分为上下文与生成标记,仅选择与生成标记关联最强、语义最代表性的一类上下文标记进行缓存。该策略有效减少冗余注意力计算,同时保持编辑一致性和保真度。实验表明,HetCache在常用基础模型上实现显著加速,达到2.67倍延迟降低和浮点运算量减少,且编辑质量几乎无损。

原文摘要 · Abstract (English)

Diffusion-based video editing has emerged as an important paradigm for high-quality and flexible content generation. However, despite their generality and strong modeling capacity, Diffusion Transformers (DiT) remain computationally expensive due to the iterative denoising process, posing challenges for practical deployment. Existing video diffusion acceleration methods primarily exploit denoising timestep-level feature reuse, which mitigates the redundancy in denoising process, but overlooks the architectural redundancy within the DiT that many attention operations over spatio-temporal tokens are redundantly executed, offering little to no incremental contribution to the model output. This work introduces HetCache, a training-free diffusion acceleration framework designed to exploit the inherent heterogeneity in diffusion-based masked video-to-video (MV2V) generation and editing. Instead of uniformly reuse or randomly sampling tokens, HetCache assesses the contextual relevance and interaction strength among various types of tokens in designated computing steps. Guided by spatial priors, it divides the spatial-temporal tokens in DiT model into context and generative tokens, and selectively caches the context tokens that exhibit the strongest correlation and most representative semantics with generative ones. This strategy reduces redundant attention operations while maintaining editing consistency and fidelity. Experiments show that HetCache achieves a noticeable acceleration, including a 2.67$\times$ latency speedup and FLOPs reduction over commonly used foundation models, with negligible degradation in editing quality.

扩散模型视频生成加速推理缓存优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。