用指数竞争法提升大模型推理速度，理论分析更严谨。
Speculative Sampling via Exponential Races
- 通过指数竞争机制生成候选文本，加速大模型推理
- 推导出生成速度与草案生成词数的理论上限关系
- 方法性能达当前最优，适合高效推理场景
推测解码通过使用较小的草稿模型加速大语言模型的推理。本文揭示了推测解码与信道模拟之间令人惊讶的联系——后者旨在以最少比特数模拟有噪信道。这一联系使我们能够对推测解码所能实现的速度提升进行信息论分析。基于此，我们推导出当草稿模型生成 $k$ 个词时，生成速度与 $k$ 的显式关系，该关系为所有 $k$ 提供了上界。此外，我们提出一种新的指数竞争推测解码方法（ERSD），其性能达到当前最优水平。
原文摘要 · Abstract (English)
Speculative decoding accelerates large language model inference using a smaller draft model. In this paper, we establish a surprising connection between speculative decoding and channel simulation, which aims at simulating a noisy channel using as few bits as possible. This connection allows us to provide an information-theoretic analysis of the speed up that can be achieved by speculative decoding. Leveraging this link, we derive an explicit relation between generation speed-up and the number of tokens $k$ generated by the draft model for large $k$, which serves as an upper bound for all $k$. We also propose a novel speculative decoding method via exponential race ERSD that matches state-of-the-art performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。