通过近邻匹配优化跳层策略，提升自推测解码的跨领域速度
KNN-SSD: Enabling Dynamic Self-Speculative Decoding via Nearest Neighbor Layer Set Optimization
- 用近邻搜索动态选择跳过的网络层，适配不同输入领域
- 在多个模型和任务上实现1.3倍至1.6倍的推理加速
- 无需额外参数或训练，适合追求高效推理的部署场景
推测解码（Speculative Decoding, SD）已成为加速大语言模型（LLM）推理的主流方法，可在不降低生成质量的前提下提升效率。其核心是利用轻量模型快速生成多个候选词元，并通过目标LLM并行验证。自推测解码（Self-Speculative Decoding）通过跳过部分网络层构建草稿模型，避免引入额外参数或训练开销。然而，本工作发现该方法对领域变化高度敏感，导致加速性能显著下降。为此，我们提出KNN-SSD，利用K-近邻（KNN）搜索为不同领域的输入动态匹配最优跳层组合。在多种模型和任务上的实验表明，该方法可带来1.3x–1.6x的推理加速。
原文摘要 · Abstract (English)
Speculative Decoding (SD) has emerged as a widely used paradigm to accelerate the inference of large language models (LLMs) without compromising generation quality. It works by efficiently drafting multiple tokens using a compact model and then verifying them in parallel using the target LLM. Notably, Self-Speculative Decoding proposes skipping certain layers to construct the draft model, which eliminates the need for additional parameters or training. Despite its strengths, we observe in this work that drafting with layer skipping exhibits significant sensitivity to domain shifts, leading to a substantial drop in acceleration performance. To enhance the domain generalizability of this paradigm, we introduce KNN-SSD, an algorithm that leverages K-Nearest Neighbor (KNN) search to match different skipped layers with various domain inputs. We evaluated our algorithm in various models and multiple tasks, observing that its application leads to 1.3x-1.6x speedup in LLM inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。