通过统一多草稿与块验证，提升语言模型推理速度。
SpecTr-GBV: Multi-Draft Block Verification Accelerating Speculative Decoding
- 将多草稿与贪婪块验证融合为统一框架，优化验证过程。
- 在五个数据集上实现更高加速比与块效率，保持输出质量。
- 理论证明可达到理想接受长度上限，适合高并发推理场景。
自回归语言模型因序列解码导致推理延迟高。推测解码（SD）通过轻量级草稿模型生成候选词，再由大模型选择性验证来缓解此问题。现有方法或采用多草稿策略提高接受率，或使用块验证技术联合验证多个词，但二者常孤立使用。本文提出SpecTr-GBV，一种将多草稿与贪婪块验证（GBV）统一于单一框架的新方法。通过将验证步骤建模为草稿与目标词块间的最优传输问题，SpecTr-GBV同时提升理论效率与实际性能。我们理论证明，在独立同分布草稿生成假设下，SpecTr-GBV可达到该框架内可实现的最优期望接受长度，且随草稿数量增加而提升。实验在五个数据集与四个基线上评估，结果表明该方法在保持输出质量的同时，显著提升加速比与块效率。此外，我们进行了全面消融实验，分析各超参数影响。
原文摘要 · Abstract (English)
Autoregressive language models suffer from high inference latency due to their sequential decoding nature. Speculative decoding (SD) mitigates this by employing a lightweight draft model to propose candidate tokens, which are selectively verified by a larger target model. While existing methods either adopt multi-draft strategies to increase acceptance rates or block verification techniques to jointly verify multiple tokens, they remain limited by treating these improvements in isolation. In this work, we propose SpecTr-GBV, a novel SD method that unifies multi-draft and greedy block verification (GBV) into a single framework. By formulating the verification step as an optimal transport problem over draft and target token blocks, SpecTr-GBV improves both theoretical efficiency and empirical performance. We theoretically prove that SpecTr-GBV achieves the optimal expected acceptance length physically attainable within the framework of i.i.d. draft generation, and this bound improves as the number of drafts increases. Empirically, we evaluate SpecTr-GBV across five datasets and four baselines. Our method achieves superior speedup and significantly higher block efficiency while preserving output quality. In addition, we perform comprehensive ablation studies to evaluate the impact of various hyperparameters in the model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。