用轻量结构提升长文本推理速度，精度损失小于2%。
VSPrefill: Vertical-Slash Sparse Attention with Lightweight Indexing for Long-Context Prefilling
- 通过垂直斜线模式构建稀疏注意力，无需修改主干参数。
- 128k上下文下速度提升4.95倍，精度保留98.35%。
- 适合需要高速长文本处理的部署场景。
大语言模型在预填充阶段的自注意力存在二次复杂度瓶颈，阻碍长上下文推理。现有稀疏注意力方法在上下文适应性、采样开销与微调成本间难以平衡。本文提出VSPrefill，一种仅需轻量训练的机制，利用注意力分布中的垂直斜线结构。其紧凑的VSIndexer模块基于加入RoPE的键值表示，预测垂直列与斜对角的上下文感知重要性得分，构建线性复杂度的稀疏掩码，无需修改骨干参数。推理时，采用自适应累积阈值策略按层分配稀疏预算，并通过融合内核实现动态索引合并。在Qwen3-4B-Instruct和LLaMA-3.1-8B-Instruct上，于LongBench与RULER基准测试中，该方法在128k上下文长度下保持98.35%全注意力精度，平均提速4.95倍，确立了准确率与效率的新帕累托前沿。
原文摘要 · Abstract (English)
The quadratic complexity of self-attention during the prefill phase impedes long-context inference in large language models. Existing sparse attention methods face a trade-off among context adaptivity, sampling overhead, and fine-tuning costs. We propose VSPrefill, a mechanism requiring lightweight training that uses the vertical-slash structural pattern in attention distributions. Our compact VSIndexer module predicts context-aware importance scores for vertical columns and slash diagonals from key-value representations augmented with RoPE. This approach constructs sparse masks with linear complexity without modifying the backbone parameters. During inference, an adaptive cumulative-threshold strategy allocates sparsity budgets per layer, while a fused kernel executes attention with on-the-fly index merging. Evaluated on Qwen3-4B-Instruct and LLaMA-3.1-8B-Instruct across the LongBench and RULER benchmarks, VSPrefill preserves 98.35% of the full attention accuracy while delivering a 4.95x average speedup at a context length of 128k. These results establish a new Pareto frontier in the trade-off between accuracy and efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。