让任意模型当草稿生成器，无损加速大模型推理
Accelerating LLM Inference with Lossless Speculative Decoding Algorithms for Heterogeneous Vocabularies
- 突破词汇表限制，任意现成模型可作草稿生成器
- 在摘要、编程等任务上最高提速2.8倍，结果无损
- 无需训练或修改，开箱即用，适合实际部署
加速大语言模型（LLM）推理是生成式AI的关键挑战。推测解码（SD）方法通过单次目标前向传播生成多个词元，实现显著效率提升。但现有方法要求草稿模型与目标模型共享同一词汇表，限制了可用草稿模型范围，常需从头训练草稿模型。本文提出三种新SD方法，打破这一词汇表约束。所有方法均保持目标分布不变（即无损），且可直接使用现成模型，无需额外训练或修改。实证表明，在摘要、编程和长上下文任务中，算法相比标准自回归解码最高提速2.8倍。该工作使任何现成模型均可作为草稿生成器，极大拓展了SD框架的实用性。
原文摘要 · Abstract (English)
Accelerating the inference of large language models (LLMs) is a critical challenge in generative AI. Speculative decoding (SD) methods offer substantial efficiency gains by generating multiple tokens using a single target forward pass. However, existing SD approaches require the drafter and target models to share the same vocabulary, thus limiting the pool of possible drafters, often necessitating the training of a drafter from scratch. We present three new SD methods that remove this shared-vocabulary constraint. All three methods preserve the target distribution (i.e., they are lossless) and work with off-the-shelf models without requiring additional training or modifications. Empirically, on summarization, programming, and long-context tasks, our algorithms demonstrate significant speedups of up to 2.8x over standard autoregressive decoding. By enabling any off-the-shelf model to serve as a drafter and requiring no retraining, this work substantially broadens the applicability of the SD framework in practice.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。