AdaCluster通过自适应聚类加速视频生成，不降质还提速四倍。
AdaCluster: Adaptive Query-Key Clustering for Sparse Attention in Video Generation
- 按查询向量角度相似性聚类，提升压缩率；键向量用欧氏相似性聚类。
- 在A40上实测速度提升1.67至4.31倍，画质几乎无损。
- 无需训练，适配不同层的异构令牌分布，适合高效视频生成部署。
视频扩散变换器（DiTs）因注意力复杂度为二次方，推理延迟极高。现有稀疏注意力方法或忽略语义相似性，或无法适应各层令牌分布差异，导致模型性能下降。我们提出AdaCluster，一种免训练的自适应聚类框架，在保持精度的同时加速DiTs生成。该方法对查询向量采用角度相似性保持聚类以实现更高压缩率，对键向量设计欧氏相似性保持聚类，包含聚类数量分配、阈值自适应聚类及高效关键聚类选择机制。在CogVideoX-2B、HunyuanVideo和Wan-2.1上于单张A40 GPU上的实验表明，速度最高提升4.31倍，质量损失可忽略。
原文摘要 · Abstract (English)
Video diffusion transformers (DiTs) suffer from prohibitive inference latency due to quadratic attention complexity. Existing sparse attention methods either overlook semantic similarity or fail to adapt to heterogeneous token distributions across layers, leading to model performance degradation. We propose AdaCluster, a training-free adaptive clustering framework that accelerates the generation of DiTs while preserving accuracy. AdaCluster applies an angle-similarity-preserving clustering method to query vectors for higher compression, and designs a euclidean-similarity-preserving clustering method for keys, covering cluster number assignment, threshold-wise adaptive clustering, and efficient critical cluster selection. Experiments on CogVideoX-2B, HunyuanVideo, and Wan-2.1 on one A40 GPU demonstrate up to 1.67-4.31x speedup with negligible quality degradation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。