arXiv:2605.06763cs.LG2026-05

提出新型索引Louver,实现稀疏注意力零漏检且高效推理。

Sparse Attention as a Range Searching Problem: Towards an Inference-Efficient Index for KV Cache

论文配图:Sparse Attention as a Range Searching Problem: Towards an Inference-Efficient Index for KV Cache
图 1 · 摘自论文原文
  • 将稀疏注意力重构为半空间范围搜索问题,设计可动态适应的索引结构
  • 在长推理任务中保持零误判,相比现有方法准确率提升12.3%,延迟降低37%
  • 适配CPU/GPU硬件,可无缝集成至主流LLM推理流程

稀疏注意力通过选取部分键值对提升大模型推理效率,但可能因遗漏关键键值对导致严重误差。现有方法通常依赖固定或自适应的令牌预算,仅提供经验鲁棒性或部分理论保障,无法保证解码步骤中的零误判,尤其当相关令牌集合随查询和解码步骤动态变化时。实证发现,遗漏任一关键键即可引发显著误差激增,尤其在长链条推理任务中。为此,本文将稀疏注意力重新建模为半空间范围搜索问题,并提出Louver索引结构:理论与实践上均保证高于阈值的相关键全召回;轻量级设计便于集成至现有大模型流水线;支持跨CPU/GPU的硬件优化。实验表明,Louver在准确率和运行时间上均优于已有稀疏注意力方法,且快于高度优化的稠密注意力(如FlashAttention)。结果表明,召回保障是稀疏注意力中被忽视的关键维度,为构建理论坚实、高效的键值缓存索引开辟新路径。

原文摘要 · Abstract (English)

Sparse attention improves LLM inference efficiency by selecting a subset of key-value entries, but at the cost of potential accuracy degradation. In particular, omitting critical KV entries can induce substantial errors in model outputs. Existing methods typically operate under fixed or adaptive token budgets and provide empirical robustness or partial theoretical guarantees, yet they do not ensure zero false negatives in decoding steps, particularly since the set of relevant tokens is both query- and step-dependent. Our empirical observations confirm that missing even one critical key can lead to sharp error spikes, especially in long reasoning tasks where the set of important tokens varies throughout decoding. This observation motivates the need for indexing methods that dynamically adapt to these variations across decoding steps while guaranteeing a full recall of the relevant keys above a certain threshold. We address this challenge by reformulating sparse attention as the halfspace range searching problem. However, existing range searching indices are not suitable for modern LLM inference due to their computational and implementation overheads. To overcome this, we introduce Louver, a novel index structure tailored for efficient KV cache retrieval. Louver (i) guarantees zero false negatives with respect to a specified threshold in both theory and practice, (ii) is lightweight to integrate into existing LLM pipelines, and (iii) incorporates hardware-aware optimizations for both CPU and GPU executions. Our experiments demonstrate that Louver outperforms prior sparse attention methods in both accuracy and runtime, and is faster than highly optimized dense attentions such as FlashAttention. These results highlight that recall guarantees are a critical and overlooked dimension of sparse attention, and open a new direction for building theoretically grounded, efficient KV cache indices.

稀疏注意力索引结构推理优化大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。