arXiv:2603.12038cs.LGcs.AI2026-03

不训练模型,通过稳定注意力记忆加速长文本生成。

Slow-Fast Inference: Training-Free Inference Acceleration via Within-Sentence Support Stability

  • 将生成分为高频轻量快步和低频密集慢步,用稀疏记忆复用减少计算。
  • 在长上下文场景下提速1.6到14.4倍,质量接近全量注意力基线。
  • 无需重新训练,可直接部署于现有模型,适合长推理任务使用。

长上下文自回归解码仍成本高昂,因每一步都需重复处理不断增长的历史。我们观察到:在句子内或短语语义连贯段落中,主要注意力支持通常保持稳定。受此启发,提出无需训练的Slow-Fast Inference(SFI)框架,将生成分解为频繁的低成本快步与偶尔的密集注意力慢步。快步复用紧凑的稀疏记忆以实现高效解码;慢步在语义边界附近触发,模型回溯更广上下文,并用选择器刷新后续快步所用的记忆。在不同上下文长度下,SFI实现了约1.6×至14.4×的吞吐提升,在长上下文与长思维链设置中普遍保持与全量键值(full-KV)基线相当的质量。由于SFI无需训练且可直接应用于现有检查点,为当前自回归推理模型在长上下文、长时序及智能体工作负载中降低推理开销提供了实用路径。

原文摘要 · Abstract (English)

Long-context autoregressive decoding remains expensive because each decoding step must repeatedly process a growing history. We observe a consistent pattern during decoding: within a sentence, and more generally within a short semantically coherent span, the dominant attention support often remains largely stable. Motivated by this observation, we propose Slow-Fast Inference (SFI), a training-free decoding framework that decouples generation into frequent low-cost fast steps and occasional dense-attention slow steps. Fast steps reuse a compact sparse memory for efficient decoding. Slow steps are triggered near semantic boundaries. At slow steps, the model revisits the broader context and uses the Selector to refresh the selected memory for subsequent fast steps. Across the evaluated context lengths, SFI delivers approximately $1.6\times$--$14.4\times$ higher decoding throughput while generally maintaining quality on par with the full-KV baseline across long-context and long-CoT settings. Because SFI is training-free and applies directly to existing checkpoints, it offers a practical path to reducing inference cost for contemporary autoregressive reasoning models in long-context, long-horizon, and agentic workloads.

推理加速注意力机制自回归生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。