arXiv:2602.03152cs.CL2026-02中稿 · ICLR被引 5

通过频段敏感的稀疏注意力,高效压缩长文本处理中的键值缓存。

FASA: Frequency-aware Sparse Attention

  • 利用旋转位置编码的频段级稀疏性,动态判断关键词元。
  • 仅保留256个词元即可接近全量缓存性能,缓存使用量降为18.9%。
  • 适合长文本建模与复杂推理任务,显著提升推理速度。

大型语言模型在处理长输入时面临键值缓存内存开销过大的瓶颈。现有词元剪枝方法或静态保留易导致信息丢失,或依赖启发式规则无法捕捉查询相关的词元重要性。本文提出FASA框架,通过动态预测词元重要性实现查询感知的词元剔除。其核心发现是:旋转位置编码(RoPE)在频段块(FC)层面存在功能性稀疏性,少数“主导”频段块始终与完整注意力头具有高度上下文一致性,可作为无成本的显著词元代理指标。FASA首先利用主导频段块识别关键词元,再仅对剪枝后的子集进行聚焦注意力计算。在从序列建模到复杂思维链推理的多种长上下文任务中,FASA持续优于所有基线方法,达到近似最优性能,且在约束预算下表现稳健。在LongBench-V1上,仅保留256个词元即达近100%全缓存性能;在AIME24上,仅用18.9%缓存便实现2.56倍加速。

原文摘要 · Abstract (English)

The deployment of Large Language Models (LLMs) faces a critical bottleneck when handling lengthy inputs: the prohibitive memory footprint of the Key Value (KV) cache. To address this bottleneck, the token pruning paradigm leverages attention sparsity to selectively retain a small, critical subset of tokens. However, existing approaches fall short, with static methods risking irreversible information loss and dynamic strategies employing heuristics that insufficiently capture the query-dependent nature of token importance. We propose FASA, a novel framework that achieves query-aware token eviction by dynamically predicting token importance. FASA stems from a novel insight into RoPE: the discovery of functional sparsity at the frequency-chunk (FC) level. Our key finding is that a small, identifiable subset of "dominant" FCs consistently exhibits high contextual agreement with the full attention head. This provides a robust and computationally free proxy for identifying salient tokens. Building on this insight, FASA first identifies a critical set of tokens using dominant FCs, and then performs focused attention computation solely on this pruned subset. Across a spectrum of long-context tasks, from sequence modeling to complex CoT reasoning, FASA consistently outperforms all token-eviction baselines and achieves near-oracle accuracy, demonstrating remarkable robustness even under constraint budgets. Notably, on LongBench-V1, FASA reaches nearly 100\% of full-KV performance when only keeping 256 tokens, and achieves 2.56$\times$ speedup using just 18.9\% of the cache on AIME24.

稀疏注意力长文本建模键值缓存优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。