无需训练即可提升稀疏视频生成效率,通过误差感知路由优化注意力补偿。
SVG-EAR: Parameter-Free Linear Compensation for Sparse Video Generation via Error-aware Routing
- 基于语义聚类的中心点近似跳过的注意力块,实现无参数补偿。
- 在万像2.2和混元视频上分别提速1.77倍和1.93倍,保持29.759和31.043的高PSNR。
- 引入误差感知路由,优先精确计算误差大的区块,适合追求高效生成的开发者。
扩散Transformer(DiTs)已成为视频生成的主流骨干网络,但其二次方复杂度的注意力机制仍是主要瓶颈。稀疏注意力通过仅计算部分注意力块来降低开销,但现有方法或直接丢弃其余块导致信息损失,或依赖可学习预测器逼近,带来训练成本与输出分布偏移。本文发现:经语义聚类后,每块内的键值具有强相似性,可用少量聚类中心点有效概括。基于此,提出SVG-EAR——一种无参数线性补偿分支,用中心点近似跳过的块以恢复贡献。尽管中心补偿对多数块准确,但少数块仍会失败。传统稀疏化依据注意力得分选择块,却无法反映补偿误差大小。SVG-EAR采用轻量级探针估计每块补偿误差,精确计算误差与代价比最高的块,同时补偿跳过块。理论证明了注意力重建误差与聚类质量的关系。实验表明,该方法显著提升质量-效率权衡,在相同生成保真度下提高吞吐量。总体上,建立优于先前方法的帕累托前沿,在Wan2.2和HunyuanVideo上分别实现最高1.77×和1.93×加速,保持29.759和31.043的PSNR。
原文摘要 · Abstract (English)
Diffusion Transformers (DiTs) have become a leading backbone for video generation, yet their quadratic attention cost remains a major bottleneck. Sparse attention reduces this cost by computing only a subset of attention blocks. However, prior methods often either drop the remaining blocks, which incurs information loss, or rely on learned predictors to approximate them, introducing training overhead and potential output distribution shifting. In this paper, we show that the missing contributions can be recovered without training: after semantic clustering, keys and values within each block exhibit strong similarity and can be well summarized by a small set of cluster centroids. Based on this observation, we introduce SVG-EAR, a parameter-free linear compensation branch that uses the centroid to approximate skipped blocks and recover their contributions. While centroid compensation is accurate for most blocks, it can fail on a small subset. Standard sparsification typically selects blocks by attention scores, which indicate where the model places its attention mass, but not where the approximation error would be largest. SVG-EAR therefore performs error-aware routing: a lightweight probe estimates the compensation error for each block, and we compute exactly the blocks with the highest error-to-cost ratio while compensating for skipped blocks. We provide theoretical guarantees that relate attention reconstruction error to clustering quality, and empirically show that SVG-EAR improves the quality-efficiency trade-off and increases throughput at the same generation fidelity on video diffusion tasks. Overall, SVG-EAR establishes a clear Pareto frontier over prior approaches, achieving up to 1.77$\times$ and 1.93$\times$ speedups while maintaining PSNRs of up to 29.759 and 31.043 on Wan2.2 and HunyuanVideo, respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。