动态路由稀疏注意力，让长文本推理快6.5倍且精度损失小。
RouteSparse: Input-Conditional Pattern Routing for Budgeted Long-Context Prefilling
- 按输入动态选择最优稀疏模式，而非固定分配
- 128K上下文下速度提升6.5倍，精度仅降0.2点
- 适合需要高效长文本处理的部署场景
动态稀疏注意力可避免长上下文预填充中的二次计算开销，而无需修改模型权重。MInference为每头预先分配一种稀疏模式，并估算该模式在所有提示中的稀疏索引。这种设计高效，但假设头的偏好模式和稀疏预算对所有输入都适用。我们提出RouteSparse，将每个注意力头与提示段落路由到一个小型、GPU高效的稀疏模式库中。通过低成本探针评估模式效用与不确定性，延迟感知路由器据此选择模式与预算，不确定情况则回退至更密集掩码。我们将路由建模为约束风险最小化问题，从被省略的概率质量推导出注意力输出误差证书，并在长上下文检索、问答、摘要和语言建模任务上进行评估。在128K token提示下，基于Llama 3.1-8B-Instruct的实验显示，RouteSparse相较稠密注意力实现6.5倍预填充加速，精度下降0.2点；而固定每头路由方案为7.3倍加速，精度下降1.6点。消融实验证明，输入条件路由、硬件性能剖析及选择性稠密回退均贡献于质量-延迟权衡。
原文摘要 · Abstract (English)
Dynamic sparse attention can reduce the quadratic cost of long-context prefilling without changing model weights. MInference assigns each attention head one pattern offline and estimates that pattern's sparse indices for every prompt. This design is efficient, but it assumes that a head's preferred pattern and sparsity budget remain suitable across inputs. We introduce RouteSparse, which routes each head and prompt segment among a small library of GPU-efficient sparse patterns. A low-cost probe estimates pattern utility and uncertainty; a latency-aware router then selects a pattern and budget, while uncertain cases fall back to a denser mask. We formulate routing as constrained risk minimization, derive an attention-output error certificate from omitted probability mass, and evaluate the method on long-context retrieval, question answering, summarization, and language modeling. On Llama 3.1-8B-Instruct with 128K-token prompts, RouteSparse achieves $6.5\times$ dense prefill speed with a 0.2-point RULER drop relative to dense attention, compared with $7.3\times$ speed and a 1.6-point drop for fixed per-head routing. Ablations confirm that input-conditional routing, hardware profiling, and selective dense fallback each contribute to the quality--latency tradeoff.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。