通过异步拆分加速大模型低延迟生成，实现超快推理。
SwiftSpec: Ultra-Low Latency LLM Decoding by Scaling Asynchronous Speculative Decoding
- 异步拆分架构让小模型与大模型并行生成，摆脱依赖瓶颈。
- 在8张Hopper卡上实现Llama3-70B每秒348词元的生成速度。
- 适合高并发低延迟场景，如聊天机器人和代码助手。
大语言模型（LLMs）的低延迟解码对聊天机器人和代码助手等应用至关重要，但在单查询场景下生成长文本仍显缓慢。现有推测解码（结合小模型与大模型）和张量并行各自提升效率，但传统方法难以同时应用，因计算不均衡、KV缓存不一致及小批量张量并行下的通信开销。本文提出SwiftSpec，一种面向超低延迟的LLM解码系统。该系统以异步、解耦方式重构推测解码流程，使各组件可灵活扩展，并将小模型开销移出关键路径。为此，提出并行树生成、树感知的KV缓存管理及融合优化的延迟敏感内核，解决上述挑战。在5个模型家族和6个数据集上，相比当前最优推测解码系统平均提速1.75倍；亮点为在8张Nvidia Hopper GPU上实现Llama3-70B每秒348词元的生成速度，成为该规模下已知最快的低延迟服务系统。
原文摘要 · Abstract (English)
Low-latency decoding for large language models (LLMs) is crucial for applications like chatbots and code assistants, yet generating long outputs remains slow in single-query settings. Prior work on speculative decoding (which combines a small draft model with a larger target model) and tensor parallelism has each accelerated decoding. However, conventional approaches fail to apply both simultaneously due to imbalanced compute requirements (between draft and target models), KV-cache inconsistencies, and communication overheads under small-batch tensor-parallelism. This paper introduces SwiftSpec, a system that targets ultra-low latency for LLM decoding. SwiftSpec redesigns the speculative decoding pipeline in an asynchronous and disaggregated manner, so that each component can be scaled flexibly and remove draft overhead from the critical path. To realize this design, SwiftSpec proposes parallel tree generation, tree-aware KV cache management, and fused, latency-optimized kernels to overcome the challenges listed above. Across 5 model families and 6 datasets, SwiftSpec achieves an average of 1.75x speedup over state-of-the-art speculative decoding systems and, as a highlight, serves Llama3-70B at 348 tokens/s on 8 Nvidia Hopper GPUs, making it the fastest known system for low-latency LLM serving at this scale.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。