arXiv:2505.12398cs.CLcs.AI2025-05NeurIPS被引 7

提出叶子到根的验证新方法,提升大模型生成速度与效率。

Traversal Verification for Speculative Tree Decoding

  • 从叶子节点向根节点逆序验证,避免提前丢弃有效候选序列。
  • 在多个模型和任务上,接受长度和吞吐量均显著优于现有方法。
  • 理论证明结果与目标模型分布一致,实现无损加速,适合高效推理场景。

推测解码是加速大语言模型的一种有前景的方法。其核心思想是使用轻量级草稿模型推测目标模型在未来多个时间步的输出,并并行验证这些推测的标记。为提高接受率,现有框架通常在每个时间步构建包含多个候选标记的标记树。然而,它们依赖于标记级别的验证机制,带来两个关键局限:首先,序列的概率分布不同于单个标记的概率分布,导致最优接受长度不理想;其次,当前验证方案从根节点开始,逐层自顶向下进行。一旦父节点被拒绝,所有子节点也必须丢弃,造成推测候选的低效利用。本文提出遍历验证(Traversal Verification),一种通过自底向上(叶子到根)遍历重新思考验证范式的新型推测解码算法。我们的方法考虑从当前节点到根节点的完整标记序列的接受性,并保留可能有效的子序列,而这些子序列在现有方法中会被过早丢弃。我们理论证明,通过遍历验证获得的概率分布与目标模型完全一致,确保无损推断的同时实现显著加速。实验结果表明,在不同大语言模型和多项任务上,该方法始终优于现有方法,显著提升接受长度和吞吐量。

原文摘要 · Abstract (English)

Speculative decoding is a promising approach for accelerating large language models. The primary idea is to use a lightweight draft model to speculate the output of the target model for multiple subsequent timesteps, and then verify them in parallel to determine whether the drafted tokens should be accepted or rejected. To enhance acceptance rates, existing frameworks typically construct token trees containing multiple candidates in each timestep. However, their reliance on token-level verification mechanisms introduces two critical limitations: First, the probability distribution of a sequence differs from that of individual tokens, leading to suboptimal acceptance length. Second, current verification schemes begin from the root node and proceed layer by layer in a top-down manner. Once a parent node is rejected, all its child nodes should be discarded, resulting in inefficient utilization of speculative candidates. This paper introduces Traversal Verification, a novel speculative decoding algorithm that fundamentally rethinks the verification paradigm through leaf-to-root traversal. Our approach considers the acceptance of the entire token sequence from the current node to the root, and preserves potentially valid subsequences that would be prematurely discarded by existing methods. We theoretically prove that the probability distribution obtained through Traversal Verification is identical to that of the target model, guaranteeing lossless inference while achieving substantial acceleration gains. Experimental results across different large language models and multiple tasks show that our method consistently improves acceptance length and throughput over existing methods.

大模型推理推测解码加速生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。