提出在线高效稀疏注意力，实现长序列建模的线性开销与高质量表现。
$π$-Attention: Online Efficient Sparse Transformers for Long-Context Modeling
- 动态维护局部邻居与长程索引，通过自适应先验融合注意力
- 在4K-32K上下文长度上逼近稠密注意力性能，每步计算复杂度为O(k)
- 适合需要长序列建模且对推理效率要求高的场景
稀疏注意力对长序列Transformer至关重要，它限制每个词元仅关注有限邻域，从而降低全自注意力的二次复杂度。局部窗口能有效捕捉近邻上下文，但在中等深度下会因感受野受限而影响长距离建模。本文提出π-Attention，一种在线高效的稀疏注意力算子：当词元逐次到达时，每一步维持一个包含局部邻居和π-索引长程获取的流式工作集，并通过共享softmax下的自适应先验进行融合。不同于预先生成全局稀疏掩码，π-Attention在实时工作集上计算注意力，具备层次感知的输入输出优化。我们分析了因果可达性与最小深度，证明每步代价保持O(k)。在语言建模、Long Range Arena及效率评估上的实验表明，在4K至32K上下文长度下，其性能持续优于局部窗口及其他稀疏基线，接近稠密注意力质量，实现线性开销。
原文摘要 · Abstract (English)
Sparse attention is crucial in long-context Transformers, which restricts each token to a limited neighborhood and thereby reduces the quadratic cost of full self-attention. Local windows capture nearby context effectively, yet they induce a receptive-field bottleneck for dependencies beyond the window, limiting long-range modeling under moderate depth. In this paper, we propose $π$-Attention, an \emph{online efficient} sparse attention operator: as tokens arrive, each step maintains a streaming working set of local neighbors plus a $π$-indexed long-range fetch, fused by an adaptive prior under a shared softmax. Rather than materializing a global sparse mask in advance, $π$-Attention computes attention on the live working set with hierarchy-aware IO. We analyze causal reachability and minimum depth under this online rule, and show per-step cost remains $\mathcal{O}(k)$. Experiments on language modeling, Long Range Arena, and efficiency profiling---across 4K--32K context lengths---show consistent gains over local-window and other sparse baselines, approaching dense attention quality at linear cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。