用冷门模型做热门模型的草稿,大幅提速大模型推理。
SPECTRE: Hybrid Ordinary-Parallel Speculative Serving for Resource-Efficient LLM Inference

- 将冷门模型作为远程草稿生成器,与目标模型并行验证。
- 在大模型部署中实现最高2.28倍加速,较最优基线提升66%。
- 适合需要高效推理的大模型服务场景,代码已开源。
大型语言模型服务系统正逐渐成为多模型云平台,用户需求呈现长尾分布:少数热门大模型接收大部分请求,而众多小型尾模型则长期处于低利用率状态。本文提出SPECTRE(并行推测解码的多租户远程草稿框架),通过推测解码将未充分利用的小型模型服务复用于高负载大模型的服务中,生成草稿并行于目标模型验证。SPECTRE采用三种技术提升并行效率:基于吞吐量分析阈值的混合普通-并行推测解码策略、保持草稿-目标重叠的推测优先调度机制,以及降低草稿延迟的提示压缩方法。我们在SGLang中实现了SPECTRE,评估了多种草稿-目标模型组合、推理基准、真实长上下文工作负载及不同批量大小。结果表明,SPECTRE在显著提升大模型服务吞吐量的同时,仅对尾模型原生任务造成轻微干扰。在大模型部署中,包括Qwen3-235B-A22B(TP=8)场景,SPECTRE相较自回归解码最高提速2.28倍,并在推测解码基线基础上额外提升66%。代码已开源:https://github.com/sgl-project/sglang/pull/22272。
原文摘要 · Abstract (English)
LLM serving platforms are increasingly deployed as multi-model cloud systems, where user demand is often long-tailed: a few popular large models receive most requests, while many smaller tail models remain underutilized. We propose \textbf{SPECTRE} (Parallel \textbf{SPEC}ulative Decoding with a Multi-\textbf{T}enant \textbf{RE}mote Drafter), a serving framework that reuses underutilized tail-model services as remote drafters for heavily loaded large-model services through speculative decoding. SPECTRE enables draft generation and target-side verification to run in parallel, and makes such parallelism effective through three techniques: a hybrid ordinary-parallel speculative decoding strategy guided by a threshold derived from throughput analysis, speculative priority scheduling to preserve draft--target overlap under multi-tenant traffic, and draft-side prompt compression to reduce draft latency. We implement SPECTRE in \texttt{SGLang} and evaluate it across multiple draft--target model pairs, reasoning benchmarks, real-world long-context workloads, and a wide range of batch sizes. Results show that SPECTRE consistently improves large-model serving throughput while causing only minor interference to the native workloads of tail-model services. In large-model deployments, including Qwen3-235B-A22B with TP=8, SPECTRE achieves up to \textbf{2.28$\times$ speedup} over autoregressive decoding and up to an additional \textbf{66\% relative improvement} over the strongest speculative decoding baselines. Talk is cheap, we show you the code: https://github.com/sgl-project/sglang/pull/22272.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。