通过稀疏自推测解，加速大模型推理，提升吞吐量达2.13倍。
Accelerating Large-Scale Reasoning Model Inference with Sparse Self-Speculative Decoding
- 用稀疏注意力机制复用验证阶段信息，智能筛选关键生成步骤。
- 在多种模型和数据集上实现最高2.13倍的吞吐量提升。
- 适合需要高效长序列推理的场景，如复杂逻辑任务部署。
推理型语言模型在复杂任务中通过生成详尽的思维链（CoT）展现出强大能力。然而，这种长序列生成使推理瓶颈从计算转向内存。每生成一个新标记，模型需对所有已生成标记进行全量注意力计算，导致不断增长的键值缓存（KV-Cache）内存访问压力。为解决此问题，我们提出SparseSpec，一种基于自推测的推测解码框架。该框架使用同一模型作为草稿与目标模型（即自推测），并引入新颖的稀疏注意力机制PillarAttn，通过巧妙复用验证阶段信息精准选择关键标记。同时，SparseSpec协同设计三项系统优化：（1）统一调度器，批量处理标记草稿与验证；（2）延迟验证以实现CPU/GPU重叠；（3）动态KV-Cache管理，最大化内存利用率。在多个模型与数据集上，SparseSpec超越现有最优方案，最高实现2.13倍吞吐量提升。
原文摘要 · Abstract (English)
Reasoning language models have demonstrated remarkable capabilities on challenging tasks by generating elaborate chain-of-thought (CoT) solutions. However, such lengthy generation shifts the inference bottleneck from compute-bound to memory-bound. To generate each token, the model applies full attention to all previously generated tokens, requiring memory access to an increasingly large KV-Cache. Consequently, longer generations demand more memory access for every step, leading to substantial pressure on memory bandwidth. To address this, we introduce SparseSpec, a speculative decoding framework that reuses the same model as the draft and target models (i.e., self-speculation). SparseSpec features a novel sparse attention mechanism, PillarAttn, as the draft model, which accurately selects critical tokens via elegantly reusing information from the verification stage. Furthermore, SparseSpec co-designs self-speculation with three system innovations: (1) a unified scheduler to batch token drafting and verification, (2) delayed verification for CPU/GPU overlap, and (3) dynamic KV-Cache management to maximize memory utilization. Across various models and datasets, SparseSpec outperforms state-of-the-art solutions, with an up to 2.13x throughput speedup.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。