破解推测解码中候选词被接受的机制,提升实际推理效率
When Is a Draft Accepted? A Theory of Acceptance in Speculative Decoding

- 构建基于目标模型分布的拒绝区域理论框架
- 发现松弛与树形结构可显著扩大可认证接受范围
- 适用于贪心解码、熵阈值等真实系统中的常见策略
推测解码通过快速起草器生成候选词,由大模型验证以加速语言模型推理。现有理论多聚焦于随机、分布保持场景,而实际系统常采用贪心解码、宽松接受规则或树状候选集,其成功依赖局部排序与阈值判断。本文建立此类情形的理论框架,识别多数接受准则的拒绝区域为目标分布的下水平集。针对严格贪心、加法/乘法松弛、前m项松弛及熵阈值接受等策略,给出精确KL散度证书与紧致边际边界。进一步扩展至贪心树解码,推导出目标最优词始终被前m项覆盖的精确与仅基于边际的证书。在Qwen3模型上的实验表明,松弛与树结构显著扩大了可认证接受区域,尤其在目标分布边际较小时效果更优。该工作补充了现有分布保持分析,刻画了实践中普遍存在的确定性局部接受事件。
原文摘要 · Abstract (English)
Speculative decoding accelerates language model inference by using a fast drafter to propose candidate tokens that are then verified by a larger target model. Existing theory largely studies the stochastic, distribution-preserving setting, where the goal is to exactly sample from the target distribution. In contrast, many practical systems use greedy decoding, relaxed acceptance rules, or tree-based candidate sets, where success is governed by local ranking and threshold events rather than exact distributional equality. We develop a theory for these regimes. We identify that many common acceptance criteria have rejection regions that can be characterized as lower level sets of the target distribution. For these, we characterize the exact KL divergence required for rejection yielding exact certificates and sharp margin-based bounds for strict greedy decoding, additive and multiplicative relaxed acceptance, top-(m) relaxed criteria, and entropy-thresholded acceptance. We then extend the framework to greedy tree decoding, deriving exact and margin-only certificates for when the target greedy token remains covered by the drafter's top-(m) candidates. Finally, we evaluate the resulting certificates on Qwen3 models, showing that relaxed and tree-based criteria substantially enlarge the region of certified acceptance, especially on decoding steps with low target model distribution margin. These results complement existing distribution-preserving analyses of speculative decoding by characterizing the deterministic local acceptance events common in practical inference systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。