构建首个统一的推测解码评估基准,真实反映生产环境性能差异。
SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding

- 设计语义多样数据集与吞吐量导向测试集,覆盖高并发到低延迟场景。
- 发现合成数据会高估实际吞吐,最优草稿长度依赖批大小。
- 适配vLLM/TensorRT-LLM等生产引擎,适合部署优化者使用。
推测解码(SD)已成为加速大语言模型推理的关键技术。由于其性能高度依赖输入数据,现有基准普遍存在任务多样性不足、缺乏吞吐量评估支持,且依赖高层实现,无法反映真实生产环境。为此,我们提出SPEED-Bench,一个统一的评估基准,涵盖多样语义领域和真实服务场景。该基准包含精心筛选的定性数据集(Qualitative data split),强调数据样本的语义多样性;同时提供吞吐量数据集(Throughput data split),支持从低并发延迟敏感场景到高负载吞吐导向场景的性能评估。通过集成vLLM和TensorRT-LLM等生产级引擎,SPEED-Bench能揭示传统基准掩盖的系统行为。我们量化发现:合成输入会显著高估真实世界吞吐,最优草稿长度随批大小变化,低多样性数据存在偏差,且先进草稿生成器中的词汇剪枝策略存在潜在缺陷。SPEED-Bench已开源,旨在建立推测解码算法实用对比的统一标准。
原文摘要 · Abstract (English)
Speculative Decoding (SD) has emerged as a critical technique for accelerating Large Language Model (LLM) inference. Unlike deterministic system optimizations, SD performance is inherently data-dependent, meaning that diverse and representative workloads are essential for accurately measuring its effectiveness. Existing benchmarks suffer from limited task diversity, inadequate support for throughput-oriented evaluation, and a reliance on high-level implementations that fail to reflect production environments. To address this, we introduce SPEED-Bench, a comprehensive suite designed to standardize SD evaluation across diverse semantic domains and realistic serving regimes. SPEED-Bench offers a carefully curated Qualitative data split, selected by prioritizing semantic diversity across the data samples. Additionally, it includes a Throughput data split, allowing speedup evaluation across a range of concurrencies, from latency-sensitive low-batch settings to throughput-oriented high-load scenarios. By integrating with production engines like vLLM and TensorRT-LLM, SPEED-Bench allows practitioners to analyze system behaviors often masked by other benchmarks. We highlight this by quantifying how synthetic inputs overestimate real-world throughput, identifying batch-size dependent optimal draft lengths and biases in low-diversity data, and analyzing the caveats of vocabulary pruning in state-of-the-art drafters. We release SPEED-Bench to establish a unified evaluation standard for practical comparisons of SD algorithms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。