SpecBlock通过分块迭代生成候选文本,显著降低大模型推理开销。
SpecBlock: Block-Iterative Speculative Decoding with Dynamic Tree Drafting

- 分块迭代生成依赖路径,每轮生成多个连续词元,减少调用次数。
- 在44-52%的起草成本下,平均提速8-13%,最高达19%。
- 适合追求高吞吐、低延迟的大模型部署场景。
推测解码通过生成候选延续树并在一次目标前向中验证来加速大语言模型推理。现有草案生成器分为两类:自回归式(如EAGLE-3)保持路径依赖但每层需调用一次起草器,导致起草耗时占比高;并行式草案生成器虽可一次预测多个位置,但缺乏相互感知,导致验证器拒绝率高。本文提出SpecBlock,一种分块迭代草案生成器,每轮生成K个依赖位置组成一个块,通过重复块扩展构建草案树。为维持路径依赖,块内采用逐层状态转移,跨块则允许从任意前一块位置继续,继承隐藏状态。为优化验证预算分配,引入联合训练的排名头动态调整分支策略。针对训练中从未生成前缀的问题,使用有效前缀掩码抑制错误位置的损失。此外,部署时引入代价感知的贝叶斯机制,仅在预期吞吐提升超过更新成本时才更新草案器。实验表明,SpecBlock在44-52%的起草成本下相比EAGLE-3实现8-13%的平均提速,代价感知适应进一步将优势扩大至11-19%。
原文摘要 · Abstract (English)
Speculative decoding accelerates LLM inference by drafting a tree of candidate continuations and verifying it in one target forward. Existing drafters fall into two camps with opposite weaknesses. Autoregressive drafters such as EAGLE-3 preserve dependence along each draft path but call the drafter once per tree depth, making drafting a non-trivial share of per-iteration latency. Parallel drafters cut drafter calls by predicting multiple future positions in one forward, but each position is predicted without seeing the others, producing paths the verifier rejects. In this paper, we propose SpecBlock, a block-iterative drafter that combines path dependence with cheap drafting. Each drafter forward produces K dependent positions and we call this a block. The draft tree grows through repeated block expansions. Two mechanisms explicitly carry path dependence to keep later draft positions accurate. Within each block, a layer-wise shift carries the previous position's hidden state into every decoder layer. Across blocks, each new block can start from any position of the previous block, inheriting its hidden state to extend the path. To spend verifier budget where acceptance is likely, a co-trained rank head replaces the fixed top-k tree by allocating per-position branching during drafting. To avoid training the drafter on prefixes it never produces at inference, a valid-prefix mask drops the loss at later positions once an earlier one is wrong. Beyond static drafting, a cost-aware bandit at deployment uses free verifier feedback to update the drafter selectively, only when the expected throughput gain exceeds the update cost. Experiments show that SpecBlock improves mean speedup by 8-13% over EAGLE-3 at 44-52% of its drafting cost, and cost-aware adaptation extends this lead to 11-19%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。