通过识别注意力头的动态差异,提升视频生成的效率与质量。
HEART: Exploiting Head Heterogeneity in Sparse Attention for Video Diffusion

- 根据头的时序稳定性决定是否复用稀疏掩码,减少冗余计算。
- 按每个头对稀疏化的敏感度分配不同阈值,提升整体精度。
- 无需重训练,可直接接入现有稀疏注意力框架,适合视频生成研究者。
稀疏注意力通过让每个注意力头仅关注少量交互来加速视频扩散过程。现有方法已基于输入构建头特定的稀疏模式,但我们发现这些头在两个较隐蔽但关键的方面存在差异:其一,某些头的稀疏掩码在多个去噪步骤中保持稳定,而另一些则快速变化;其二,各头对稀疏化程度的敏感性差异显著,相同阈值可能导致最终去噪速度误差大不相同。忽略这些差异会导致冗余掩码预测和全局稀疏预算下的次优阈值校准。我们提出HEART(Heterogeneity-Exploiting Adaptive Refresh and Thresholding),一种无需训练的框架,以利用这两种头异质性。首先,时间掩码复用(TMR)通过轻量级每头查询-键漂移信号判断缓存掩码是否可靠,仅当漂移超过阈值时才刷新。其次,误差引导的预算校准(EBC)离线评估候选阈值,基于频率加权的去噪速度误差为每个头分配合适阈值,满足全局稀疏预算。HEART无需重训练、权重修改或稀疏核变更,可直接集成至现有稀疏注意力流程。在Wan2.1-1.3B、Wan2.1-14B和HunyuanVideo-13B上,HEART持续将XAttention和SVG2等先进稀疏注意力方法的质量-效率边界向外推进。
原文摘要 · Abstract (English)
Sparse attention accelerates video diffusion by allowing each attention head to focus on only a small subset of interactions. Existing methods already construct head-specific sparse patterns conditioned on the input. However, we find that these heads also differ in two less obvious but practically important ways. First, for some heads, sparse attention masks remain stable across many denoising steps, whereas others change rapidly. Second, heads differ substantially in their sensitivity to sparsification: applying the same threshold can induce markedly different errors in the final denoising velocity. Ignoring these differences leads to redundant mask prediction and suboptimal threshold calibration across heads under a global sparsity budget. We present HEART, short for Heterogeneity-Exploiting Adaptive Refresh and Thresholding, a training-free framework that exploits both forms of head heterogeneity. First, Temporal Mask Reuse (TMR) uses a lightweight per-head query-key drift signal to determine whether a cached sparse mask remains reliable across denoising steps, refreshing it only when the drift exceeds a prescribed threshold. Second, Error-guided Budgeted Calibration (EBC) evaluates candidate thresholds offline using a frequency-weighted denoising-velocity error, and assigns each head an appropriate threshold under a global sparsity budget. HEART requires no retraining, weight modification, or sparse-kernel changes, and can be integrated directly into existing sparse-attention pipelines. Across Wan2.1-1.3B, Wan2.1-14B, and HunyuanVideo-13B, HEART consistently pushes the quality--efficiency frontier of advanced sparse attention methods such as XAttention and SVG2 outward.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。