arXiv:2608.26112cs.CL2026-08

用多个不同成本的草稿模型协作构建树状草稿,提升推理速度与质量。

TreeGraft: Adaptive Multi-Drafter Grafting for Tree-Based Speculative Decoding

  • 引入多草稿模型协同构建共享树状草稿,动态调度强弱模型分工。
  • 在10组模型和6个基准上平均提速15.1%,最高达26.6%。
  • 适合追求高效推理的LLM部署场景,尤其对资源敏感应用友好。

推测解码通过先草稿后验证的范式加速大语言模型推理。在此基础上,树结构方法通过组织候选路径提升可接受长度。然而,现有方法使用单一草稿模型,面临权衡:小型草稿模型虽快但生成质量低,大型草稿模型质量高但延迟大。为此,我们提出TreeGraft,一种多草稿框架,让不同成本的草稿模型共同构建共享草稿树。TreeGraft利用更强的草稿模型重评分候选、重新选择嫁接位置,并恢复未探索的潜力路径。同时,非破坏性整合强草稿扩展,保留可能被目标模型接受的现有分支。为控制成本,引入轻量级调度器,基于离线价值系统蒸馏决策何时调用强草稿模型。在10个模型对和6个基准测试中,TreeGraft平均优于两个固定单草稿终点策略中的更优者15.1%,最大提升达26.6%。代码已开源。

原文摘要 · Abstract (English)

Speculative decoding accelerates large language model inference through a draft-then-verify paradigm. Building on this, tree-structured methods improve inference by organizing proposals into multiple candidate paths, increasing the accepted length. However, existing tree-structured methods use a single drafter for all drafting steps, creating a dilemma: a smaller drafter is fast but yields lower-quality trees, whereas a larger drafter improves tree quality but suffers from high latency. To address this, we propose TreeGraft, a multi-drafter framework in which drafters of different costs jointly construct a shared draft tree. TreeGraft uses the stronger drafter to rescore candidates by updating scores assigned by the weaker drafter, reselect grafting positions, and recover promising paths left unexplored. It also integrates stronger drafter expansions non-destructively, preserving existing branches that may still be accepted by the target model. Together, these designs improve the quality of the shared draft tree. To control the drafting cost, TreeGraft introduces a lightweight scheduler distilled from an offline value system to decide when to call the stronger drafter. Across 10 model pairs and 6 benchmarks, TreeGraft outperforms the better of the two fixed single-drafter endpoint strategies by 15.1% on average, reaching a maximum gain of 26.6%. Our code is available at https://github.com/fjm9933/TreeGraft.

推理加速树结构多草稿扩散模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。