arXiv:2604.21221cs.CVcs.LG2026-04被引 6

通过可训练的稀疏注意力机制,实现视频生成的低延迟与高质量长序列输出。

Sparse Forcing: Native Trainable Sparse Attention for Real-time Autoregressive Diffusion Video Generation

论文配图:Sparse Forcing: Native Trainable Sparse Attention for Real-time Autoregressive Diffusion Video Generation
图 1 · 摘自论文原文
  • 基于视觉块的局部稀疏模式,设计可学习的动态稀疏注意力机制。
  • 在5秒生成上提升VBench 0.26分,最长1分钟生成速度提升27%。
  • 适合追求实时生成与高画质的视频扩散模型开发者使用。

我们提出Sparse Forcing,一种用于自回归视频扩散模型的训练与推理范式,可在提升长时序生成质量的同时降低解码延迟。该方法基于自回归扩散推演中的实证观察:注意力集中在一组持续存在的显著视觉块上,在键值缓存中形成隐式的时空记忆,并在滑动窗口内呈现局部结构化的块稀疏模式。在此基础上,我们提出一种可训练的原生稀疏机制,能够学习压缩、保留并更新这些持久性块,同时将每窗计算限制在动态选择的局部邻域内。为实现大规模训练与推理的实用性,进一步设计了高效的GPU内核Persistent Block-Sparse Attention(PBSA),加速稀疏注意力与内存更新,实现低延迟、内存高效解码。实验表明,Sparse Forcing在5秒文本到视频生成中相较Self-Forcing提升VBench 0.26分,解码速度提升1.11–1.17倍,峰值KV缓存占用降低42%;在更长时序生成中优势更显著,20秒和1分钟生成分别实现VBench提升+0.68与+2.74,速度提升1.22倍与1.27倍。

原文摘要 · Abstract (English)

We introduce Sparse Forcing, a training-and-inference paradigm for autoregressive video diffusion models that improves long-horizon generation quality while reducing decoding latency. Sparse Forcing is motivated by an empirical observation in autoregressive diffusion rollouts: attention concentrates on a persistent subset of salient visual blocks, forming an implicit spatiotemporal memory in the KV cache, and exhibits a locally structured block-sparse pattern within sliding windows. Building on this observation, we propose a trainable native sparsity mechanism that learns to compress, preserve, and update these persistent blocks while restricting computation within each local window to a dynamically selected local neighborhood. To make the approach practical at scale for both training and inference, we further propose Persistent Block-Sparse Attention (PBSA), an efficient GPU kernel that accelerates sparse attention and memory updates for low-latency, memory-efficient decoding. Experiments show that Sparse Forcing improves the VBench score by +0.26 over Self-Forcing on 5-second text-to-video generation while delivering a 1.11-1.17x decoding speedup and 42% lower peak KV-cache footprint. The gains are more pronounced on longer-horizon rollouts, delivering improved visual quality with +0.68 and +2.74 VBench improvements, and 1.22x and 1.27x speedups on 20-second and 1-minute generations, respectively.

视频生成扩散模型稀疏注意力实时推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。