通过修剪与检索协同,实现高速推理且不损失生成质量。
Draft Less, Retrieve More: Hybrid Tree Construction for Speculative Decoding
- 先剪枝后补全,用检索填补剪枝留下的空缺
- 短文本场景下最高提速5.41倍,大模型平均提速21.8%
- 无需训练,适配长文本与大模型部署
推测解码(SD)通过先生成草稿再验证的范式加速大语言模型推理。为提升接受率,现有方法构建密集草稿树,但带来严重显存带宽和计算开销,制约整体速度。动态深度剪枝虽能降低延迟,却会丢弃潜在有效候选,无法达到密集树的接受率上限。本文发现:从密集到剪枝的转变释放了大量计算预算。为此提出Graft框架,将剪枝与检索设计为相互增强的操作——剪枝释放预算供检索使用,检索补偿剪枝带来的覆盖损失并恢复已接受长度。采用顺序‘剪枝-补全’机制,将高预测性检索词插入剪枝产生的空位,以近零开销填补拓扑缺口。Graft完全无需训练且无损。全面评估表明,其在短文本、长文本及大规模模型场景下均建立新帕累托前沿。短文本基准上最高提速5.41倍,对大模型Qwen3-235B的平均速度提升较EAGLE-3高出21.8%。还初步探索了Graft在DFlash式块草稿范式中的应用,为扩展至非自回归草稿树提供初步证据与见解。
原文摘要 · Abstract (English)
Speculative decoding (SD) accelerates large language model inference by leveraging a draft-then-verify paradigm. To maximize the acceptance rate, recent methods construct expansive draft trees, which unfortunately incur severe VRAM bandwidth and computational overheads that bottleneck end-to-end speedups. While dynamic-depth pruning can reduce this latency by removing marginal branches, it also discards potentially valid candidates, preventing the acceptance rate from reaching the upper bound of dense trees. In this paper, we identify a critical opportunity in resource allocation: the transition from dense to pruned drafting frees up significant computational budget. To break this Pareto tradeoff, we introduce Graft, a compensation framework that couples pruning and retrieval as mutually reinforcing operations. Pruning supplies sufficient budget for retrieval, while retrieval compensates for pruning-induced coverage loss and recovers accepted length. By employing a sequential `prune-then-graft' mechanism, Graft attaches highly predictive retrieved tokens into positions opened by pruning, filling the topological gaps with near-zero overhead. Graft is entirely training-free and lossless. Comprehensive evaluations show that Graft establishes a new Pareto frontier across practical deployment settings, including short-context generation, long-context generation, and large-scale models. On short-context benchmarks, it achieves up to 5.41$\times$ speedup and improves average speedup over EAGLE-3 by up to 21.8% on the large-scale Qwen3-235B. We also provide a preliminary exploration of applying Graft to the DFlash-style block drafting paradigm, offering initial evidence and insights for extending grafting beyond autoregressive draft trees.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。