通过批量并行设计，让模型推理时的预测与验证重叠，大幅提速。
MineDraft: A Framework for Batch Parallel Speculative Decoding
- 用两个请求批次交替进行预测和验证，实现并行处理
- 吞吐量最高提升75%，端到端延迟降低39%
- 可直接作为vLLM插件部署，适合生产环境
推测解码（SD）通过使用小型草稿模型生成候选词元，再由大型目标模型验证，以加速大语言模型推理。然而，标准SD常受限于生成与验证阶段的严格串行执行。为此，本文提出MineDraft，一种批并行推测解码（PSD）框架，通过重叠不同批次的生成与验证过程，有效隐藏生成延迟。理论分析表明，PSD相比标准SD效率显著更高。MineDraft通过创新的批并行设计，维持两个请求批次，使一个批次的生成与另一批次的验证并行进行。实验结果表明，该方法在吞吐量上较标准SD最高提升75%，端到端延迟最高降低39%。此外，我们已将MineDraft实现为vLLM的插件，证明其适用于生产级推理系统。
原文摘要 · Abstract (English)
Speculative decoding (SD) accelerates large language model inference by using a smaller draft model to propose draft tokens that are subsequently verified by a larger target model. However, the performance of standard SD is often limited by the strictly sequential execution of these drafting and verification stages. To address this, this paper proposes MineDraft, a batch parallel speculative decoding (PSD) framework designed to effectively hide drafting latency by overlapping it with verification. Our theoretical analysis shows that PSD is substantially more efficient than standard SD. MineDraft realizes the PSD through a novel batch-parallel design that maintains two batches of requests, overlapping drafting for one batch with verification for the other. Our experimental results show significant improvements of \alg{} in both throughput (up to 75%) and end-to-end latency (up to 39%) over standard SD. Furthermore, we have implemented MineDraft as a plugin for vLLM, demonstrating its practicality for production-ready inference systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。