用词频组合加速自回归生成，无需训练即可提速
The N-Grammys: Accelerating Autoregressive Inference with Learning-Free Batched Speculation
- 用模型权重和上下文提取N-gram作为无学习的候选生成策略
- 在小k值下，真实生成词常位于候选列表前几位，实现高效验证
- 无需预处理或修改主模型，可直接插入现有推理流程
推测解码通过较小的草稿模型并行生成候选词，以加速语言模型的自回归生成。本文探索了无需学习、成本极低的草稿策略，即从模型权重和上下文提取的N-gram。尽管基础模型预测的下一个词很少是这些简单策略的最优预测，但通常出现在其前-k个候选中（k较小）。基于此，我们证明多种简单策略的组合可在不同任务上实现显著的推理加速。整体性能与复杂方法相当，且无需昂贵预处理或修改主模型，支持无缝‘即插即用’集成到现有推理管道中。
原文摘要 · Abstract (English)
Speculative decoding aims to speed up autoregressive generation of a language model by verifying in parallel the tokens generated by a smaller draft model.In this work, we explore the effectiveness of learning-free, negligible-cost draft strategies, namely $N$-grams obtained from the model weights and the context. While the predicted next token of the base model is rarely the top prediction of these simple strategies, we observe that it is often within their top-$k$ predictions for small $k$. Based on this, we show that combinations of simple strategies can achieve significant inference speedups over different tasks. The overall performance is comparable to more complex methods, yet does not require expensive preprocessing or modification of the base model, and allows for seamless `plug-and-play' integration into pipelines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。