提出高效视频生成的稀疏注意力方法,显著降低延迟并提升画质。
HyperVAttention: Efficient Sparse Attention with Spatio-Temporal Clustering for Video Diffusion

- 基于时空局部聚类与轻量级更新策略,减少聚类开销。
- 端到端延迟降低2.13倍,画质优于现有无训练稀疏注意力方法。
- 适合追求高效长视频生成的研究者与开发者使用。
视频扩散变换器(VDiTs)在高保真视频生成方面表现突出,但其生成长时视频的能力受限于自注意力机制的二次复杂度。现有基于聚类的稀疏注意力方法虽改善了质量-速度权衡,但仍受两大瓶颈制约:聚类开销大,以及由不规则聚类块导致的计算纹理阵列(CTA)利用率低。本文提出无需训练的稀疏注意力框架HyperVAttention(HVA),从两方面协同优化:首先引入3D局部窗口聚类,利用视频帧间时空局部性,将聚类中心搜索限制在固定局部邻域,并通过定制Triton核实现高效执行;其次设计混合聚类策略,仅在关键步骤进行完整聚类,中间步骤仅更新部分令牌,利用去噪过程中的聚类稳定性避免冗余计算。为提升CTA利用率,提出面向硬件的聚类合并策略,通过并行凝聚式合并最小化对齐执行成本,提高块密度与近似精度,充分利用空闲瓦片资源。实验表明,HVA在文本到视频生成任务中建立了无训练稀疏注意力的新帕累托前沿,端到端延迟最高降低2.13倍,同时在保真度上优于现有基线方法。
原文摘要 · Abstract (English)
Video Diffusion Transformers (VDiTs) have demonstrated significant capabilities in high-fidelity video generation. However, their ability to produce long-duration videos is fundamentally constrained by the quadratic complexity of the self-attention mechanism. Recent clustering-based sparse attention methods improve the quality-speed trade-off by grouping semantically similar tokens, but their practical efficiency remains limited by two bottlenecks: substantial clustering overhead and low CTA utilization caused by irregular cluster-induced blocks. We propose HyperVAttention (HVA), a training-free sparse attention framework that addresses both bottlenecks jointly. To reduce clustering overhead, we introduce 3D local-window clustering, which exploits the spatio-temporal locality of video tokens to restrict centroid search to fixed local neighborhoods, and implement it with a custom Triton kernel for efficient execution. We further propose a hybrid clustering strategy that performs full clustering only at anchor steps and updates only subset tokens at intermediate steps, leveraging the temporal stability of cluster assignments across denoising steps. To improve CTA utilization, we present hardware-aware cluster merging that minimizes CTA-aligned execution cost through parallel agglomerative merging, improving block density and approximation fidelity by utilizing idle tile capacity. Together, these components reduce clustering overhead, avoid redundant updates, and better align sparse attention with the fixed tile structure of modern GPU kernels. Experiments on Text-to-Video generation show that HVA establishes a new Pareto frontier for training-free sparse attention in video diffusion, reducing end-to-end latency by up to $2.13\times$ while improving fidelity over existing training-free sparse attention baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。