用密集检索提升大模型推测解码效率，显著加快生成速度。
DReSD: Dense Retrieval for Speculative Decoding
- 用上下文嵌入的近似最近邻搜索替代传统字符串匹配
- 接受率提升87%，单次接受令牌长度增加65%
- 适合追求推理加速的AI系统开发者
推测解码（SD）通过使用高效的小模型预先提出后续若干词元，并由大语言模型在一次前向传播中验证，从而降低延迟并保持输出质量。本文聚焦于基于检索的推测解码，其中小模型从非参数化数据存储中检索下一个词元。当前主流方法稀疏检索（REST）依赖字符串表面形式匹配，因上下文短且严格匹配而效果受限。为此，我们提出密集检索用于推测解码（DReSD），利用上下文化的词元嵌入进行近似最近邻搜索，以获取最语义相关的词元序列。大量实验表明，相较于稀疏检索（REST），DReSD平均实现87%更高的接受率、65%更长的可接受词元长度以及19%更快的生成速度。
原文摘要 · Abstract (English)
Speculative decoding (SD) accelerates Large Language Model (LLM) generation by using an efficient draft model to propose the next few tokens, which are verified by the LLM in a single forward call, reducing latency while preserving its outputs. We focus on retrieval-based SD where the draft model retrieves the next tokens from a non-parametric datastore. Sparse retrieval (REST), which operates on the surface form of strings, is currently the dominant paradigm due to its simplicity and scalability. However, its effectiveness is limited due to the usage of short contexts and exact string matching. Instead, we introduce Dense Retrieval for Speculative Decoding (DReSD), a novel framework that uses approximate nearest neighbour search with contextualised token embeddings to retrieve the most semantically relevant token sequences for SD. Extensive experiments show that DReSD achieves (on average) 87% higher acceptance rates, 65% longer accepted tokens and 19% faster generation speeds compared to sparse retrieval (REST).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。