为专家混合模型设计动态推测解码,避免性能下降并提升吞吐量。
Utility-Driven Speculative Decoding for Mixture-of-Experts
- 根据推测收益与验证开销比动态启停推测,避免无效计算
- 在五种主流MoE模型上实现7-14%吞吐提升,慢速降低至5%
- 适配代码、数学等多任务场景,适合高并发推理服务部署
GPU内存带宽是低延迟大语言模型推理的主要瓶颈。推测解码通过轻量级草稿模型提出K个词元,由主模型并行验证,从而提升词元吞吐量。在传统密集模型中,每次迭代需加载全部权重,推测无额外延迟开销。而新兴的专家混合(MoE)模型每词元仅激活部分权重,显著减少数据移动。然而我们发现,推测在MoE中效果不佳:草稿词元整体激活更多权重,导致验证时间增加2-3倍。当吞吐增益无法抵消开销时,推理速度反而下降达1.5倍,使推测不可行。即使有效,最优K值也随任务、模型及请求间动态变化。为此,我们提出Cascade框架,基于推测效用(收益/成本比)选择性启用推测,并动态调整K值。该框架在短测试阶段评估效用,若低于1则禁用推测;高于1时测试多个K值,选取最大化效用的配置用于长执行阶段。我们在vLLM中实现该框架,在五种主流MoE模型上对代码、数学、抽取及混合任务进行评估,结果表明其将最差慢速控制在5%(相比1.5x),吞吐量相较静态K提升7-14%,使推测解码在MoE中变得可行。
原文摘要 · Abstract (English)
GPU memory bandwidth is the main bottleneck for low-latency Large Language Model (LLM) inference. Speculative decoding leverages idle GPU compute by using a lightweight drafter to propose K tokens, which the LLM verifies in parallel, boosting token throughput. In conventional dense LLMs, all model weights are fetched each iteration, so speculation adds no latency overhead. Emerging Mixture of Experts (MoE) models activate only a subset of weights per token, greatly reducing data movement. However, we show that speculation is ineffective for MoEs: draft tokens collectively activate more weights, increasing data movement and verification time by 2-3x. When token throughput gains fail to offset this overhead, speculation causes slowdowns up to 1.5x, making it infeasible. Even when useful, the optimal K varies by task, model, and even between requests and iterations. Thus, despite widespread use in dense LLMs, speculation remains impractical in leading MoEs. We present Cascade, a utility-driven framework that selectively enables speculation to avoid slowdowns and dynamically tunes K to accelerate MoE serving. Cascade uses a lightweight metric, speculation utility, the ratio of token gains to verification cost, which shows iteration-level locality, enabling periodic decisions via short test and longer set phases. For each request, Cascade disables speculation if utility drops below one during testing, and when utility exceeds one, tests multiple K-values to choose the utility-maximizing K for the set phase. We implement Cascade in vLLM and evaluate it on five popular MoEs with workloads spanning code, math, extraction, and mixed tasks. Cascade limits slowdown to 5% (vs. 1.5x) and improves throughput by 7-14% over static K, making speculative decoding practical for MoEs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。