仅用数百步训练,就能让大模型实现高效稀疏推理。
Full Attention Strikes Back: Transferring Full Attention into Sparse within Hundred Training Steps

- 发现大模型本就有内在稀疏性,只需少量调整即可转为稀疏结构。
- 在100万上下文长度下,预填充速度提升9.36倍,解码速度提升2.01倍。
- 适合追求高效长文本处理且不想重训的开发者使用。
大语言模型在长上下文推理中受限于全注意力机制的二次计算开销。现有高效替代方案要么依赖原生稀疏训练,要么采用启发式令牌淘汰策略,导致效率、训练成本与准确率之间存在权衡。本文指出,全注意力模型本身已具备内在稀疏性,可通过极小适应实现高度稀疏化。基于三点观察:(1)仅有少数注意力头真正需要完整长上下文处理;(2)长距离信息检索主要由低维子空间决定,可用16维索引器高效定位相关令牌;(3)有效令牌预算强烈依赖查询,动态top-$p$选择优于固定top-$k$稀疏化。据此提出RTPurbo方法,仅对检索头保留完整键值缓存,并引入轻量级令牌索引器实现稀疏注意力。利用模型内在稀疏性,RTPurbo仅需数百次训练即可完成稀疏化。在长上下文基准和推理任务上的实验表明,该方法保持近无损精度的同时带来显著效率提升,包括100万上下文长度下最高达9.36倍的预填充加速和约2.01倍的解码加速。结果表明,无需昂贵的原生稀疏预训练,即可从标准全注意力训练模型获得强稀疏推理能力。
原文摘要 · Abstract (English)
Long-context inference in large language models is bottlenecked by the quadratic cost of full attention. Existing efficient alternatives often rely either on native sparse training or on heuristic token eviction, creating an undesirable trade-off among efficiency, training cost, and accuracy. In this work, we show that full-attention LLMs are already intrinsically sparse and can be transformed into highly sparse models with only minimal adaptation. Our approach is built on three observations: (1) only a small subset of attention heads truly requires full long-context processing; (2) long-range retrieval is governed primarily by a low-dimensional subspace, allowing relevant tokens to be retrieved efficiently with a 16-dimensional indexer; and (3) the useful token budget is strongly query-dependent, making dynamic top-$p$ selection more suitable than fixed top-$k$ sparsification. Based on these insights, we propose RTPurbo, which retains the full KV cache only for retrieval heads and introduces a lightweight token indexer for sparse attention. By exploiting the model's intrinsic sparsity, RTPurbo achieves sparsification with only a few hundred training steps. Experiments on long-context benchmarks and reasoning tasks show that RTPurbo preserves near-lossless accuracy while delivering substantial efficiency gains, including up to a 9.36$\times$ prefill speedup at 1M context and about a 2.01$\times$ decode speedup. These results suggest that strong sparse inference can be obtained from standard full-attention training without expensive native sparse pretraining.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。