动态调整生成块长度，让大模型推理更快更准
PACER: Blockwise Pre-verification for Speculative Decoding with Adaptive Length
- 用轻量预验证层按块检测生成内容是否合理
- 实测比传统方法快2.66倍，结合Ouroboros可达3.09倍
- 适合追求推理速度的部署场景，尤其大模型应用
推测解码（SD）是一种在不牺牲准确性的前提下加速大语言模型推理的强大技术。通常，SD使用小型草稿模型生成固定数量的草稿标记，并由目标模型并行验证。然而，我们的实验表明，最优草稿长度在不同解码步骤间差异显著，固定长度限制了进一步提速潜力。为此，我们提出Pacer，一种通过轻量可训练预验证层动态控制草稿长度的新方法。该层在发送至目标模型前逐块预验证草稿标记，若块级验证失败，草稿模型即停止生成。我们在多个SD模型对上实现Pacer，并在多种基准测试中评估其性能。结果表明，Pacer相比自回归解码最高提升2.66倍，且始终优于标准推测解码。当与Ouroboros结合时，最高提速达3.09倍。
原文摘要 · Abstract (English)
Speculative decoding (SD) is a powerful technique for accelerating the inference process of large language models (LLMs) without sacrificing accuracy. Typically, SD employs a small draft model to generate a fixed number of draft tokens, which are then verified in parallel by the target model. However, our experiments reveal that the optimal draft length varies significantly across different decoding steps. This variation suggests that using a fixed draft length limits the potential for further improvements in decoding speed. To address this challenge, we propose Pacer, a novel approach that dynamically controls draft length using a lightweight, trainable pre-verification layer. This layer pre-verifies draft tokens blockwise before they are sent to the target model, allowing the draft model to stop token generation if the blockwise pre-verification fails. We implement Pacer on multiple SD model pairs and evaluate its performance across various benchmarks. Our results demonstrate that Pacer achieves up to 2.66x Speedup over autoregressive decoding and consistently outperforms standard speculative decoding. Furthermore, when integrated with Ouroboros, Pacer attains up to 3.09x Speedup.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。