用反对角线和快速判断注意力块重要性,实现高效长文本推理。
XAttention: Block Sparse Attention with Antidiagonal Scoring

- 通过计算注意力矩阵反对角线之和,快速评估块重要性。
- 在多个长序列任务上实现最高13.5倍的计算加速,精度接近全注意力。
- 无需修改模型结构,可直接替换现有注意力模块,适合部署优化。
长上下文Transformer模型在实际应用中至关重要,但其注意力机制的二次复杂度导致计算开销巨大。块稀疏注意力通过聚焦关键区域缓解此问题,但现有方法因块重要性评估成本高而难以兼顾精度与效率。本文提出XAttention,一种即插即用的框架,显著加速Transformer的长上下文推理。其核心思想是:注意力矩阵反对角线(从左下到右上)元素之和可有效代理块重要性,从而精准识别并剪枝非关键块,实现高稀疏度与大幅加速。在多个严苛长序列基准测试中——包括语言类的RULER和LongBench、视频理解的VideoMME、视频生成的VBench——XAttention在保持全注意力相当精度的同时,带来显著计算提升,最高实现13.5倍的注意力计算加速。结果表明,XAttention有效释放了块稀疏注意力的实用潜力,为长上下文模型在真实场景中的可扩展高效部署铺平道路。代码已开源:https://github.com/mit-han-lab/x-attention。
原文摘要 · Abstract (English)
Long-Context Transformer Models (LCTMs) are vital for real-world applications but suffer high computational costs due to attention's quadratic complexity. Block-sparse attention mitigates this by focusing computation on critical regions, yet existing methods struggle with balancing accuracy and efficiency due to costly block importance measurements. In this paper, we introduce XAttention, a plug-and-play framework that dramatically accelerates long-context inference in Transformers models using sparse attention. XAttention's key innovation is the insight that the sum of antidiagonal values (i.e., from the lower-left to upper-right) in the attention matrix provides a powerful proxy for block importance. This allows for precise identification and pruning of non-essential blocks, resulting in high sparsity and dramatically accelerated inference. Across comprehensive evaluations on demanding long-context benchmarks-including RULER and LongBench for language, VideoMME for video understanding, and VBench for video generation. XAttention achieves accuracy comparable to full attention while delivering substantial computational gains. We demonstrate up to 13.5x acceleration in attention computation. These results underscore XAttention's ability to unlock the practical potential of block sparse attention, paving the way for scalable and efficient deployment of LCTMs in real-world applications. Code is available at https://github.com/mit-han-lab/x-attention.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。