用流水线并行实现更准的快速生成，延迟隐藏得更彻底。
Speculative Pipeline Decoding: Higher-Accuracy Drafting with Hidden Latency via Pipeline Parallelism
- 将大模型分段流水处理，多个词并行推进
- 多深度特征聚合预测，接受率更高，延迟可控
- 适合追求低延迟高吞吐的部署场景
推测解码（Speculative Decoding, SD）通过先生成后验证的范式加速低并发大模型推理。主流方法依赖多词预测，导致预测难度累积且生成延迟暴露。本文提出推测流水线解码（SPD），将目标大模型分为 $n$ 个流水线阶段，使单条序列的 $n$ 个词并行推进。为保持流水线饱和，流水线草稿模块（PDM）聚合多层目标特征预测下一个词，并与每个流水线步骤并发运行，实现可控制的预测难度、更高的接受率和隐藏的草稿延迟。实验表明，在中等流水线宽度下，SPD 的理论与实际速度提升均优于 EAGLE-3，而更激进的宽度仍有优化空间。代码已开源：https://github.com/yuyijiong/speculative_pipeline_decoding。
原文摘要 · Abstract (English)
Speculative Decoding (SD) accelerates low-concurrency LLM inference with a draft-then-verify paradigm. Mainstream methods, however, rely on multi-token prediction, which incurs compounding prediction difficulty and exposed draft latency. We propose Speculative Pipeline Decoding (SPD), which partitions the target LLM into $n$ pipeline stages so that $n$ tokens of a single sequence advance in parallel. To keep the pipeline saturated, a Pipeline Draft Module (PDM) aggregates multi-depth target features to predict the next token and runs concurrently with each pipeline step, yielding bounded prediction difficulty, higher acceptance, and hidden draft latency. Experiments show that SPD achieves higher theoretical and wall-clock speedup than EAGLE-3 at moderate pipeline width, while more aggressive widths still leave room for further gains. Our code is available at https://github.com/yuyijiong/speculative_pipeline_decoding
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。