通过词频优先筛选候选词，加速大词汇量语言模型生成。
FR-Spec: Accelerating Large-Vocabulary Language Models via Frequency-Ranked Speculative Sampling
- 基于词频排序压缩词汇空间，只在高频词中选候选。
- 减少75%语言模型头计算开销，平均提速1.12倍。
- 适合大词汇量模型如Llama-3-8B，提升推理效率。
推测采样作为一种重要技术，通过先生成后验证的机制，在每次前向传播中生成多个标记以加速大语言模型（LLM）的自回归生成过程。尽管现有最优方法仅使用单层和语言模型头作为草稿模型，实现了显著的层压缩，但在大词汇量模型（如拥有128,000个标记的Llama-3-8B）上，其加速效果大幅下降。为此，我们提出FR-Spec——一种基于词频排序的推测采样框架，通过词汇空间压缩优化草稿候选选择。将草稿搜索限制在词频优先的标记子集中，该方法将语言模型头计算开销降低75%，同时保证最终输出分布等价。在多个数据集上的实验表明，相比当前最优的EAGLE-2方法，平均获得1.12倍的加速。代码已公开于https://github.com/thunlp/FR-Spec。
原文摘要 · Abstract (English)
Speculative sampling has emerged as an important technique for accelerating the auto-regressive generation process of large language models (LLMs) by utilizing a draft-then-verify mechanism to produce multiple tokens per forward pass. While state-of-the-art speculative sampling methods use only a single layer and a language modeling (LM) head as the draft model to achieve impressive layer compression, their efficiency gains are substantially reduced for large-vocabulary LLMs, such as Llama-3-8B with a vocabulary of 128k tokens. To address this, we present FR-Spec, a frequency-ranked speculative sampling framework that optimizes draft candidate selection through vocabulary space compression. By constraining the draft search to a frequency-prioritized token subset, our method reduces LM Head computation overhead by 75% while ensuring the equivalence of the final output distribution. Experiments across multiple datasets demonstrate an average of 1.12$\times$ speedup over the state-of-the-art speculative sampling method EAGLE-2. Code available at https://github.com/thunlp/FR-Spec.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。