arXiv:2511.12035cs.ARcs.CV2025-11被引 1

通过挖掘潜在空间的时空相关性,实现视频生成模型的高效推理。

Timeripple: Accelerating vDiTs by Understanding the Spatio-Temporal Correlations in Latent Space

  • 利用潜在空间中时空相关的令牌特征,重用部分注意力计算结果。
  • 在4个vDiT模型上实现85%的计算量减少,视频质量损失小于0.06%。
  • 适合需要快速视频生成且对延迟敏感的应用场景。

视频生成的快速发展推动了对高质量视频合成的需求。当前主流的视频扩散变换器(vDiT)因自注意力机制导致推理延迟严重。已有研究试图通过减少冗余计算来优化注意力,但常忽略视频流中固有的时空相关性,并直接套用大语言模型的稀疏模式。本文提出一种基于潜在空间时空相关性的原理化加速方法。我们发现,vDiT中的注意力模式主要源于通道层面的空间与时间相关性。基于此,设计了一种轻量级、自适应的重用策略,通过复用空间或时间相关令牌的局部注意力分数,近似完整注意力计算。实验表明,该方法在4个vDiT模型上相比现有最佳技术实现85%的计算节省,同时保持几乎相同的视频质量(在VBench上损失低于0.06%)。

原文摘要 · Abstract (English)

The recent surge in video generation has shown the growing demand for high-quality video synthesis using large vision models. Existing video generation models are predominantly based on the video diffusion transformer (vDiT), however, they suffer from substantial inference delay due to self-attention. While prior studies have focused on reducing redundant computations in self-attention, they often overlook the inherent spatio-temporal correlations in video streams and directly leverage sparsity patterns from large language models to reduce attention computations. In this work, we take a principled approach to accelerate self-attention in vDiTs by leveraging the spatio-temporal correlations in the latent space. We show that the attention patterns within vDiT are primarily due to the dominant spatial and temporal correlations at the token channel level. Based on this insight, we propose a lightweight and adaptive reuse strategy that approximates attention computations by reusing partial attention scores of spatially or temporally correlated tokens along individual channels. We demonstrate that our method achieves significantly higher computational savings (85\%) compared to state-of-the-art techniques over 4 vDiTs, while preserving almost identical video quality ($<$0.06\% loss on VBench).

视频生成扩散模型加速推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。