用预判验证结果的方式，让推理速度提升5倍。
Speculative Speculative Decoding
- draft 模型提前预测验证结果，实现并行推测
- 实测平均快30%，最高比自回归快5倍
- 适合追求极致推理速度的部署场景
自回归解码受限于串行特性。推测解码通过快速草稿模型预测目标模型的后续词元，并用单次目标模型前向传播并行验证，成为加速推理的标准方法。然而推测解码本身仍存在推测与验证之间的串行依赖。我们提出推测性推测解码（SSD），使推测与验证操作并行化。在验证进行时，草稿模型预先预测可能的验证结果，并为这些结果提前生成推测内容。若实际验证结果在预测集中，推测可立即返回，完全消除起草开销。我们识别出三个由 SSD 引发的关键挑战，并提出相应解决方案。最终得到 Saguaro 算法。在开源推理引擎上，其实现平均比优化后的推测解码快30%，最高比自回归解码快5倍。
原文摘要 · Abstract (English)
Autoregressive decoding is bottlenecked by its sequential nature. Speculative decoding has become a standard way to accelerate inference by using a fast draft model to predict upcoming tokens from a slower target model, and then verifying them in parallel with a single target model forward pass. However, speculative decoding itself relies on a sequential dependence between speculation and verification. We introduce speculative speculative decoding (SSD) to parallelize these operations. While a verification is ongoing, the draft model predicts likely verification outcomes and prepares speculations pre-emptively for them. If the actual verification outcome is then in the predicted set, a speculation can be returned immediately, eliminating drafting overhead entirely. We identify three key challenges presented by speculative speculative decoding, and suggest principled methods to solve each. The result is Saguaro, an optimized SSD algorithm. Our implementation is on average 30% faster than optimized speculative decoding baselines and up to 5x faster than autoregressive decoding with open source inference engines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。