arXiv:2510.09883cs.CLcs.LG2025-10ACL被引 4

DELTA通过动态选择关键令牌,让长文本推理更快更准。

DELTA: Dynamic Layer-Aware Token Attention for Efficient Long-Context Reasoning

论文配图:DELTA: Dynamic Layer-Aware Token Attention for Efficient Long-Context Reasoning
图 1 · 摘自论文原文
  • 分层设计:前层全注意力,中间层选关键令牌,后层只关注选定部分。
  • 在AIME和GPQA-Diamond上精度不降,注意力计算量减少4.25倍,速度提升1.54倍。
  • 无需训练,适合部署在需要高效长文本推理的场景。

大型推理模型(LRMs)通过生成长推理链在挑战性基准上取得顶尖性能,但其推理成本主要来自解码过程,每生成一个新令牌都需关注不断增长的完整序列。一种降低延迟的方法是丢弃键值(KV)缓存中的条目,从而减少注意力计算的活跃上下文。然而,这类稀疏注意力方法因累积选择误差以及长推导过程中令牌重要性演变,在推理任务中导致严重精度下降。我们提出 extbf{DELTA},一种无需训练的稀疏注意力机制,在不牺牲模型精度的前提下提升计算效率。DELTA 将 Transformer 层分为三组:初始层使用全注意力,少量 extit{$Δ$-层} 通过聚合头级注意力得分识别重要令牌,后续稀疏注意力层仅关注所选子集。该设计在显存中保留完整 KV 缓存以保证精度,同时避免多层全注意力计算开销。在 AIME 和 GPQA-Diamond 等推理基准上,DELTA 的精度与全注意力相当或更优,注意力计算量最多减少 4.25 倍,端到端速度提升 1.54 倍。结果表明,有选择地重用中间注意力图是实现高效长上下文推理的稳健路径。代码已公开于 https://github.com/hoenza/DELTA。

原文摘要 · Abstract (English)

Large reasoning models (LRMs) achieve state-of-the-art performance on challenging benchmarks by generating long chains of intermediate steps, but their inference cost is dominated by decoding, where each new token must attend to the entire growing sequence. One approach to reduce this latency is to evict entries from the key-value (KV) cache, thereby reducing the active context used in attention computation. However, such sparse attention methods suffer from severe accuracy degradation on reasoning tasks due to cumulative selection errors and the evolving importance of tokens over long derivations. We present \textbf{DELTA}, a training-free sparse attention mechanism that improves computational efficiency without sacrificing model accuracy. DELTA partitions transformer layers into three groups: initial layers that use full attention, a small set of \emph{$Δ$-layers} that identify salient tokens via aggregated head-level attention scores, and subsequent sparse-attention layers that attend only to the selected subset. This design preserves the full KV cache in GPU memory for accuracy, while avoiding expensive full-attention computation over many layers. On reasoning benchmarks such as AIME and GPQA-Diamond, DELTA matches or surpasses full attention in accuracy, while reducing the number of attended tokens by up to $4.25\times$ and delivering $1.54\times$ end-to-end speedup. Our results show that selective reuse of intermediate attention maps offers a robust path toward efficient long-context reasoning. The code is available at https://github.com/hoenza/DELTA.

长文本推理稀疏注意力效率优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。