提出动态选词策略，让模型推理更快更准。
Speculative Decoding with a Speculative Vocabulary
- 每步动态选择候选词集，替代固定小词汇表。
- 相比EAGLE-3，平均吞吐提升最高达8.1%。
- 适合追求高效推理的生成式AI应用。
推测解码已成为加速语言模型推理的主流方法，能在输出不变的前提下大幅提升速度。其核心是使用一个小型草稿模型预测目标模型的输出。当前最优方法采用单层解码器加输出嵌入矩阵的草稿模型，但后者在最新大模型中成为瓶颈。已有工作尝试通过缩小词汇表缓解此问题，但会降低推测成功率。本文提出一种新思路：在每一步解码时动态选择词汇子集，即SpecVocab。实验表明，该方法在多种任务上可实现比EAGLE-3更长的接受长度，平均吞吐最高提升8.1%。
原文摘要 · Abstract (English)
Speculative decoding has rapidly emerged as a leading approach for accelerating language model (LM) inference, as it offers substantial speedups while yielding identical outputs. This relies upon a small draft model, tasked with predicting the outputs of the target model. State-of-the-art speculative decoding methods use a draft model comprising a single decoder layer and output embedding matrix, with the latter dominating drafting time for the latest LMs. Recent work has sought to address this output distribution bottleneck by reducing the vocabulary of the draft model. While this can improve throughput, it compromises speculation effectiveness when the target token is out-of-vocabulary. In this paper, we argue for vocabulary speculation as an alternative to a reduced vocabulary. We propose SpecVocab, an efficient and effective method that selects a vocabulary subset per decoding step. Across a variety of tasks, we show that SpecVocab can achieve a higher acceptance length than state-of-the-art speculative decoding method, EAGLE-3. Notably, this yields up to an 8.1% increase in average throughput over EAGLE-3.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。