提出新采样方法，让小词汇量模型也能高效生成大模型内容。
Out-of-Vocabulary Sampling Boosts Speculative Decoding
- 设计可生成未登录词的新采样器，解决小词表模型效率瓶颈。
- 极端压缩词表（删75%以上）后仍保持高采纳率，传统方法已失效。
- 适合追求推理加速的AI系统开发者，尤其关注低资源部署场景。
推测解码依赖快速准确的草稿模型。当前先进语言模型采用越来越大的词表，显著拖慢草稿模型速度。一种提升效率的思路是使用小词表草稿模型，但现有采样方法无法生成未登录词，导致词表大小与采纳率之间存在权衡。本文提出首个支持未登录词采样的方法——重分配草稿核（RDK），通过重新分配草稿分布质量，在不实际恢复被剪枝词元的前提下，有效恢复采纳率。RDK利用词元相似性先验，将草稿质量集中于高重叠区域。我们从数学上证明，RDK在采纳率上优于基线与最先进采样器。同时提供一阶近似实现，将重分配时间从 $O(N^2)$ 降至 $O(N)$，使大规模词表下轻量级部署成为可能。实验表明，即使在删除超过75%词表的情况下，线性时间版本的RDK仍能显著提升采纳率，而现有方法已失效。RDK为极简词表草稿模型打开了新可能。
原文摘要 · Abstract (English)
Speculative decoding relies on fast and accurate drafters. Recent state-of-the-art language models employ larger and larger vocabularies, which significantly slows down drafters. One promising approach to boost the efficiency of speculative decoding is to use drafters with smaller vocabularies. However, existing sampling methods cannot draw out-of-vocabulary tokens, creating a tradeoff between drafters' vocabulary size and acceptance rates. This paper introduces Redistributing Drafter Kernels (RDK), the first out-of-vocabulary sampler that effectively recovers acceptance rates by virtually restoring pruned target tokens. RDK leverages token-affinity priors to reallocate drafter mass towards high-overlap regions. We prove mathematically that RDK can achieve higher acceptance rates than vanilla and state-of-the-art samplers. We provide an efficient first-order approximation of RDK and prove that it reduces redistribution times from $O(N^2)$ to $O(N)$, enabling lightweight implementations for large vocabularies. Our experiments demonstrate that this linear-time RDK significantly boosts acceptance rates even after extreme pruning (removing more than 75% of the drafter's vocabulary), where existing samplers fail. RDK opens the door to extremely pruned drafters, which were previously impractical.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。