用新模型提升大批次推理速度，降低生成成本。
SPIRe: Boosting LLM Inference Throughput with Speculative Decoding
- 设计稀疏注意力+剪枝初始化+反馈记忆的混合模型
- 大批次下吞吐量提升超100%，比基线快35%以上
- 适合长上下文且请求长度差异大的场景
推测解码（SD）已被证明在小批量时可将自回归解码延迟降低2-3倍。但要提高吞吐量、降低每令牌成本，需使用大批次解码。近期研究显示，当上下文足够长且草稿模型的键值缓存稀疏时，SD也能加速大批次解码。我们提出SPIRe，一种结合静态稀疏注意力、剪枝初始化和反馈记忆的草稿模型，在大批次下使推测解码吞吐量相比使用更小草稿模型提升超过100%，相比强基线稀疏自推测提升超过35%。该方法在请求间上下文长度差异显著时尤为有效。
原文摘要 · Abstract (English)
Speculative decoding (SD) has been shown to reduce the latency of autoregressive decoding (AD) by 2-3x for small batch sizes. However, increasing throughput and therefore reducing the cost per token requires decoding with large batch sizes. Recent work shows that SD can accelerate decoding with large batch sizes too if the context is sufficiently long and the draft model's KV cache is sparse. We introduce SPIRe, a draft model that combines static sparse attention, pruned initialization, and feedback memory to increase the modeled throughput of speculative decoding by over 100% compared to speculation with a much smaller draft model and by over 35% compared to the strong baseline of sparse self-speculation. Our approach is particularly effective when context lengths vary significantly across requests.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。