不训练模型，用稀疏注意力提升长文本生成速度
A Sparse Glimpse of the Whole: Train-Free Self-Speculative Decoding
- 自动生成轻量草稿，动态稀疏化缓存避免信息丢失
- 在多任务测试中实现最高超过自回归解码的加速效果
- 适合需要高效长文本生成且无法重新训练的场景
推测解码缓解了大语言模型推理中的内存带宽瓶颈，但加速效果受草稿开销、令牌接受率和推测长度的共同制约。本文提出统一效率分析，表明当边际接受概率低于相对草稿成本时，延长推测范围反而降低加速比。基于此，我们提出SparseSpec-L：一种无需训练的自推测解码框架，用于长上下文推理。该方法直接从目标模型生成轻量草稿，利用动态稀疏化且可召回的键值缓存；在全上下文验证过程中产生的每头注意力统计信息被复用为无额外前向传播的重要度信号，使关键历史令牌可被召回，而无需永久丢弃密集缓存。在线熵控制机制根据预期逐步效率动态选择推测长度。在多个长上下文任务与模型规模上实验显示，该方法实现持续端到端加速，最高加速超过自回归解码，同时保持目标模型输出分布不变。
原文摘要 · Abstract (English)
Speculative decoding alleviates the memory-bandwidth bottleneck in large language model inference, but its acceleration is jointly constrained by drafting overhead, token acceptance, and speculation length. We present a unified efficiency analysis showing that extending the speculation horizon can reduce rather than improve speedup when the marginal acceptance probability falls below the relative drafting cost. Guided by this analysis, we introduce SparseSpec-L, a training-free self-speculative decoding framework for long-context inference. SparseSpec-L generates lightweight drafts directly from the target model using a dynamically sparsified and recallable KV cache. It recycles per-head attention statistics produced during full-context verification as a no-extra-forward importance signal, allowing critical historical tokens to be recalled without permanently discarding the dense KV cache. An online entropy-based controller further selects the speculation length according to expected step-wise efficiency. Experiments across multiple long-context tasks and model scales show consistent end-to-end acceleration, with up to speedup over autoregressive decoding while preserving the target model's output distribution.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。