通过分层流水线打破大模型推理中的阶段依赖，实现更高效加速。
PipeSpec: Breaking Stage Dependencies in Hierarchical LLM Decoding
- 采用多模型分层流水线，异步执行并轻量协调验证与回滚。
- 实测最高提速2.54倍，且在任意非零接受率下均优于传统方法。
- 适合需要高并发、多设备部署的大模型推理场景。
推测解码通过使用较小的草稿模型并行生成候选词以加速大语言模型推理。然而，现有方法受限于串行阶段依赖，难以充分利用硬件资源。本文提出PipeSpec框架，将推测解码推广至由k个模型构成的分层流水线，支持异步执行，并通过轻量级协调机制完成预测验证与回滚。我们的分析模型刻画了各流水线阶段的词元生成速率，并证明在任意非零接受率下，该方法均能保证吞吐量提升。我们进一步推导出稳态验证概率的闭式表达式，解释了流水线深度带来的实际收益。实验结果表明，PipeSpec在文本摘要和代码生成任务中，使用LLaMA 2和3模型时，最高实现2.54倍速度提升，优于当前最优方法。验证显示，流水线效率随模型深度增加而提升，为多设备系统上的大模型推理提供可扩展加速方案。
原文摘要 · Abstract (English)
Speculative decoding accelerates large language model inference by using smaller draft models to generate candidate tokens for parallel verification. However, current approaches are limited by sequential stage dependencies that prevent full hardware utilization. We present PipeSpec, a framework that generalizes speculative decoding to $k$ models arranged in a hierarchical pipeline, enabling asynchronous execution with lightweight coordination for prediction verification and rollback. Our analytical model characterizes token generation rates across pipeline stages and proves guaranteed throughput improvements over traditional decoding for any non-zero acceptance rate. We further derive closed-form expressions for steady-state verification probabilities that explain the empirical benefits of pipeline depth. Experimental results show that PipeSpec achieves up to 2.54$\times$ speedup while outperforming state-of-the-art methods. We validate PipeSpec across text summarization and code generation tasks using LLaMA 2 and 3 models, demonstrating that pipeline efficiency increases with model depth, providing a scalable approach to accelerating LLM inference on multi-device systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。