提出统一稀疏注意力机制，高效处理长文本上下文。
A Unified Sparse Attention via Multi-Granularity Compression
- 用复合标记聚合多粒度上下文信息，动态构建稀疏注意力。
- 在多种任务中达到全注意力99%精度，速度比FlashAttention快2.61倍。
- 兼容多模态且可直接作为加速插件使用，适合实际部署场景。
大语言模型在多轮对话、程序分析等应用中对长序列理解与推理的需求日益增长。然而，核心自注意力机制随序列长度呈平方级增长，构成计算瓶颈。现有稀疏注意力方法存在权衡：训练型方法成本高，无法作为通用加速插件；推理时方法常牺牲效率或跨模态泛化能力。为此，我们提出UniSparse，引入复合标记——一种聚合多粒度上下文信息的紧凑表示。基于此抽象，UniSparse通过多粒度压缩与块级选择动态构建稀疏注意力，实现高效的GPU硬件友好执行。在从合成基准到真实应用的多种模态与任务中，UniSparse持续优于当前最优稀疏注意力方法（如MInference、XAttention、FlexPrefill），在准确率上达到全注意力的≥99%，注意力计算速度最快达FlashAttention的2.61倍。
原文摘要 · Abstract (English)
Efficient long-context understanding and reasoning are increasingly vital for large language model (LLM) applications such as multi-turn dialogue and program analysis. However, the core self-attention mechanism scales quadratically with sequence length, creating a fundamental computational bottleneck. Existing sparse attention methods alleviate this issue but face trade-offs: training-based methods are costly and cannot be directly applied as acceleration plugins for other models, while inference-time methods often compromise efficiency or cross-modal generality. To address these limitations, we present UniSparse, a unified mechanism that introduces the notion of composite tokens--compact representations that aggregate multi-granularity contextual information. Building on this abstraction, UniSparse dynamically constructs sparse attention through multi-granularity compression and block-level selection, enabling efficient and hardware-friendly execution on GPU. Across multiple modalities and tasks ranging from synthetic benchmarks to real-world applications, UniSparse consistently surpasses state-of-the-art sparse attention methods (e.g., MInference, XAttention, FlexPrefill) in both accuracy and efficiency, achieving $\ge$ 99% of full-attention accuracy and up to 2.61$\times$ faster attention computation than FlashAttention.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。