arXiv:2602.16994cs.LG2026-02

提出动态延迟树扩展,让快速采样更高效。

Dynamic Delayed Tree Expansion For Improved Multi-Path Speculative Decoding

  • 延迟分支点,先走单路径再扩展树结构
  • 使基于最优传输的方法平均吞吐提升5%
  • 适合追求推理速度的模型部署场景

多路径推测解码通过使用廉价的草稿模型生成令牌树,并用验证算法接受其中一部分来加速目标模型的无损采样。尽管已有多种验证算法用于独立同分布的滚动采样,但在相同设置下的相对性能仍不明确。本文首次系统评估了不同验证策略在多个模型族、任务和采样方式下的表现,发现遍历验证始终占优,而基于最优传输(OT)的方法明显落后。分析表明,这是由于OT方法虽在草稿树根部实现高多令牌接受率,但真正关键的多令牌收益出现在草稿与目标分布偏离更深的位置。基于此洞察,我们提出延迟树扩展:先生成部分单路径草稿,推迟独立同分布分支点。该方法保持目标分布不变,并优于根节点独立同分布滚动。进一步设计动态神经选择器,从草稿与目标特征中估计最优传输方法的预期块效率,实现上下文相关的扩展决策。该选择器使SpecInfer等基于OT的方法首次超越遍历验证,在广泛模型、数据集和采样设置下实现平均5%更高的吞吐量。

原文摘要 · Abstract (English)

Multi-path speculative decoding accelerates lossless sampling from a target model by using a cheaper draft model to generate a draft tree of tokens, and then applies a verification algorithm that accepts a subset of these. While prior work has proposed various verification algorithms for i.i.d rollouts, their relative performance under matched settings remains unclear. In this work, we firstly present a systematic evaluation of verification strategies across model families, tasks, and sampling regimes, and find that Traversal Verification dominates consistently, with OT-based methods lagging far behind. Our analysis uncovers that this occurs because OT-based methods achieve high multi-token acceptance near the root of the draft tree, while multi-token gains are most impactful deeper in the draft tree, where draft and target distributions diverge. Based on this insight, we propose delayed tree expansion, which drafts a partial single path, delaying the i.i.d. branching point. We show that delayed tree expansion preserves the target distribution and improves on root-node i.i.d rollouts. Further, we develop a dynamic neural selector that estimates the expected block efficiency of optimal-transport-based verification methods from draft and target features, enabling context-dependent expansion decisions. Our neural selector allows OT-based methods like SpecInfer to outperform Traversal Verification for the first time, achieving 5% higher average throughput across a wide range of models, datasets, and sampling settings.

推理加速推测解码动态决策最优传输

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。