SPLA通过稀疏+线性注意力，高效处理长文本且不丢信息。
SPLA: Block Sparse Plus Linear Attention for Long Context Modeling
- 用二阶泰勒展开选关键块，保留重要上下文。
- 未选块压缩为递归状态，推理时零显存访问。
- 适合长文本建模，性能接近全注意力模型。
块级稀疏注意力在长序列建模中显著提升效率，但现有方法常因完全丢弃未选块而导致选择精度低和上下文累积损失。为此，我们提出稀疏加线性注意力（SPLA），利用二阶泰勒展开导出的选择度量精准识别需精确计算注意力的关键块。不同于直接丢弃其余‘长尾’块，SPLA通过残差线性注意力（RLA）模块将其压缩为紧凑的递归状态。关键在于，为避免输入输出开销，我们推导出基于减法的优化公式：将残差表示为全局与选中块线性注意力之差，从而确保推理过程中无需显式访问未选块。实验表明，SPLA在持续预训练中缩小了与密集注意力模型的性能差距，在RULER等长序列基准上表现更优，同时保持了良好的通用知识与推理能力。
原文摘要 · Abstract (English)
Block-wise sparse attention offers significant efficiency gains for long-context modeling, yet existing methods often suffer from low selection fidelity and cumulative contextual loss by completely discarding unselected blocks. To address these limitations, we introduce Sparse Plus Linear Attention (SPLA), a framework that utilizes a selection metric derived from second-order Taylor expansions to accurately identify relevant blocks for exact attention. Instead of discarding the remaining "long tail," SPLA compresses unselected blocks into a compact recurrent state via a residual linear attention (RLA) module. Crucially, to avoid IO overhead, we derive an optimized subtraction-based formulation for RLA -- calculating the residual as the difference between global and selected linear attention -- ensuring that unselected blocks are never explicitly accessed during inference. Our experiments demonstrate that SPLA closes the performance gap in continual pretraining, surpassing dense attention models on long-context benchmarks like RULER while maintaining competitive general knowledge and reasoning capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。