提出向量级稀疏注意力,加速长视频推理且保持高精度。
VecAttention: Vector-wise Sparse Attention for Accelerating Long Context Inference
- 按视频帧垂直向量动态筛选关键信息,减少冗余计算。
- 在多个视频任务上实现2.65倍推理加速,精度接近全注意力。
- 适合需要高效处理长视频的模型部署与实时应用。
基于Transformer的视频模型在处理长上下文时面临自注意力机制二次复杂度带来的巨大计算挑战。现有稀疏注意力方法虽提升效率,但多采用粗粒度模式,导致冗余计算和性能下降。本文提出新框架VecAttention,通过观察发现视频注意力图具有显著的垂直向量稀疏性,并证明该模式在准确率-稀疏度权衡上优于现有方法。基于此,VecAttention设计轻量级重要向量选择机制,降低内存访问开销,并优化向量稀疏注意力核。在视频理解(VideoMME、LongVideoBench、VCRBench)与生成(VBench)任务上的全面评估显示,相比全注意力提速2.65倍,比当前最优稀疏方法快1.83倍,且精度与全注意力相当。
原文摘要 · Abstract (English)
Long-context video understanding and generation pose a significant computational challenge for Transformer-based video models due to the quadratic complexity of self-attention. While existing sparse attention methods employ coarse-grained patterns to improve efficiency, they typically incur redundant computation and suboptimal performance. To address this issue, in this paper, we propose \textbf{VecAttention}, a novel framework of vector-wise sparse attention that achieves superior accuracy-efficiency trade-offs for video models. We observe that video attention maps exhibit a strong vertical-vector sparse pattern, and further demonstrate that this vertical-vector pattern offers consistently better accuracy-sparsity trade-offs compared with existing coarse-grained sparse patterns. Based on this observation, VecAttention dynamically selects and processes only informative vertical vectors through a lightweight important-vector selection that minimizes memory access overhead and an optimized kernel of vector sparse attention. Comprehensive evaluations on video understanding (VideoMME, LongVideoBench, and VCRBench) and generation (VBench) tasks show that VecAttention delivers a 2.65$\times$ speedup over full attention and a 1.83$\times$ speedup over state-of-the-art sparse attention methods, with comparable accuracy to full attention. Our code is available at https://github.com/anminliu/VecAttention.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。