arXiv:2605.07363cs.LGcs.AI2026-05被引 3

用轻量路由选头,让长文本推理更快更省显存。

MISA: Mixture of Indexer Sparse Attention for Long-Context LLM Inference

论文配图:MISA: Mixture of Indexer Sparse Attention for Long-Context LLM Inference
图 1 · 摘自论文原文
  • 把多头索引器变成专家池,按需激活少数头部做筛选。
  • 8个活跃头就达到Dense DSA效果,显存占用减为1/4到1/8。
  • 适合长上下文大模型部署,尤其对高精度检索任务友好。

DeepSeek稀疏注意力(DSA)通过学习的逐令牌索引器,在推理时高效选择相关前缀令牌,但其多头设计导致计算开销大。本文提出MISA(索引器混合稀疏注意力),将索引器头视为专家池,用轻量级路由器根据块级统计仅激活少数头部进行令牌评分,大幅降低每查询成本。进一步提出分层MISA,先用路由扩大候选集再用原DSA重排序,几乎完全恢复原始选择结果。在DeepSeek-V3.2和GLM-5上,使用8个活跃头即匹配密集型DSA性能,显存消耗减少8倍或4倍;在128K上下文下保持绿色针堆热力图,每层保留超过92% DSA选中令牌。单张NVIDIA H200 GPU上,自研TileLang内核实现约3.82倍速度提升。

原文摘要 · Abstract (English)

DeepSeek Sparse Attention (DSA) sets the state of the art for fine-grained inference-time sparse attention by introducing a learned token-wise indexer that scores every prefix token and selects the most relevant ones for the main attention. To remain expressive, the indexer uses many query heads (for example, 64 on DeepSeek-V3.2) that share the same selected token set; this multi-head design is precisely what makes the indexer the dominant cost on long contexts. We propose MISA (Mixture of Indexer Sparse Attention), a drop-in replacement for the DSA indexer that treats its indexer heads as a pool of mixture-of-experts. A lightweight router uses cheap block-level statistics to pick a query-dependent subset of only a few active heads, and only those heads run the heavy token-level scoring. This preserves the diversity of the original indexer pool while reducing the per-query cost from scoring every prefix token with every head to scoring it with only a handful of routed heads, plus a negligible router term computed on a small set of pooled keys. We further introduce a hierarchical variant of MISA that uses the routed pass to keep an enlarged candidate set and then re-ranks it with the original DSA indexer to recover the final selected tokens almost exactly. With only eight active heads and no additional training, MISA matches the dense DSA indexer on LongBench across DeepSeek-V3.2 and GLM-5 while running with eight and four times fewer indexer heads respectively, and outperforms HISA on average. It also preserves fully green Needle-in-a-Haystack heatmaps up to a 128K-token context and recovers more than 92% of the tokens selected by the DSA indexer per layer. Our TileLang kernel delivers roughly a 3.82 times speedup over DSA's original indexer kernel on a single NVIDIA H200 GPU.

稀疏注意力长文本推理优化大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。