首次系统评估生成推理加速技术的实际效果，发现理论与现实差距大。
Speculative Decoding: Performance or Illusion?
- 在真实部署引擎vLLM上测试多种生成推理方法
- 验证阶段耗时主导性能，接受长度随位置和数据变化大
- 揭示理论速度提升与实际表现间巨大差距，指明改进方向
生成式解码（Speculative Decoding, SD）已成为加速大语言模型推理的流行技术，但其真实效能仍不明确，因以往评估依赖研究原型和极小批量。本文首次在生产级、广泛部署的推理引擎vLLM上对SD进行系统性研究，涵盖多种变体（n-gram、EAGLE/EAGLE-3、Draft-Model、Multi-Token Prediction），覆盖不同工作负载、模型规模和批量大小。我们分析了影响SD性能的关键因素，并量化了理论最大加速比。结果表明，目标模型的验证阶段占据主要执行时间，而接受长度在输出位置、请求和数据集间差异显著。将实测性能与理论上限对比后发现，实际与理论之间存在显著差距，由此揭示出可进一步研究的新方向。
原文摘要 · Abstract (English)
Speculative decoding (SD) has become a popular technique to accelerate Large Language Model (LLM) inference, yet its real-world effectiveness remains unclear as prior evaluations rely on research prototypes and unrealistically small batch sizes. We present, to our knowledge, the first systematic study of SD on a production-grade and widely deployed inference engine (vLLM), covering multiple SD variants ($n$-gram, EAGLE/EAGLE-3, Draft-Model, Multi-Token Prediction) across diverse workloads, model scales, and batch sizes. We analyze key factors governing SD performance, and quantify a theoretical upper bound on SD speedup. Our results show that verification by the target model dominates the execution, while acceptance length varies markedly across output token positions, requests, and datasets. Comparing measured performance with theoretical bounds reveals substantial gaps between observed and theoretical upper bounds, and we leverage this observation to highlight new research opportunities that our study opens up in improving SD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。