arXiv:2506.01979cs.DCcs.AI2025-06中稿 · ICLR被引 16

通过分支并行加速大模型推理,减少等待开销。

SpecBranch: Speculative Decoding via Hybrid Drafting and Rollback-Aware Branch Parallelism

  • 借鉴处理器分支预测,用并行分支预判可能被拒绝的生成路径。
  • 在多种模型上实现1.8到4.5倍加速,差对齐模型回滚令牌减少50%。
  • 适合需要高效推理的部署场景,尤其对不匹配模型效果显著。

最近,推测解码(Speculative Decoding, SD)作为一种加速大语言模型推理的有前途技术,通过使用小型草稿模型提前提出候选标记,并与大型目标模型并行验证。然而,现有方法仍受制于串行执行,导致草稿模型与目标模型之间存在相互等待的空隙。为解决此问题,我们受现代处理器中分支预测的启发,提出新型框架 SpecBranch,以在 SD 中实现分支并行。具体而言,我们深入分析了 SD 中分支并行的潜力,认识到关键挑战在于并行化与标记回滚之间的权衡。基于此分析,我们战略性地引入并行推测分支,以提前应对可能的拒绝情况。同时,为增强并行性,我们结合隐式草稿模型置信度与显式复用目标模型特征,动态调整草稿长度。在多个模型和基准上的大量实验表明,SpecBranch 相比自回归解码实现了超过 1.8× ∼ 4.5× 的加速,并使对齐不佳模型的回滚标记减少 50%,具备实际部署可行性。

原文摘要 · Abstract (English)

Recently, speculative decoding (SD) has emerged as a promising technique to accelerate LLM inference by employing a small draft model to propose draft tokens in advance, and validating them in parallel with the large target model. However, the existing SD methods still remain fundamentally constrained by their serialized execution, which causes the mutual waiting bubbles between the draft and target models. To address this challenge, we draw inspiration from branch prediction in modern processors and propose a novel framework \textbf{SpecBranch} to unlock branch parallelism in SD. Specifically, we first take an in-depth analysis of the potential of branch parallelism in SD, and recognize that the key challenge lies in the trade-offs between parallelization and token rollback. Based on the analysis, we strategically introduce parallel speculative branches to preemptively hedge against likely rejections. Meanwhile, to enhance parallelism, we jointly orchestrate adaptive draft lengths with a hybrid combination of the implicit draft model confidence and explicit reusing of target model features. Extensive experiments across various models and benchmarks show that SpecBranch achieves over \textbf{1.8}$\times \sim$ \textbf{4.5}$\times$ speedups against the auto-regressive decoding and reduces rollback tokens by $\textbf{50}$\% for poorly aligned models, realizing its applicability for real-world deployments.

推理加速推测解码并行计算

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。