让大模型生成更快：通过语义匹配替代字面匹配，提升预测准确性。
SENSE: Semantic Embedding Navigation with Soft-gated Evaluation for Retrieval-based Speculative Decoding
- 基于目标模型隐状态进行语义检索，避免表面词形变化干扰
- 软门控验证机制实现语义等价判断，最高支持4.09个令牌的接受长度
- 适用于需要快速生成又不牺牲质量的场景，如对话系统、内容创作
推测解码（SD）通过轻量级草稿模型预提出候选词元，由目标模型并行验证，从而加速大语言模型推理，且不降低生成质量。尽管基于检索的推测解码（RSD）因其即插即用的灵活性受到青睐，但其性能受限于严格的词汇依赖，对表面形式变化敏感，导致检索与验证均易失效。为此，我们提出 SENSE（语义嵌入导航与软门控评估）。通过将检索锚定在目标模型的隐藏状态上，SENSE 实现了稳健的语义对齐，使软门控评估模块能验证语义等价性而非表面形式。为确保严谨评测，我们在统一框架下将现有方法分解为原子组件，支持细粒度的模块级对比。跨多个领域的大量实验表明，SENSE 在 LLaMA 与 Qwen 系列模型上均优于多个基线，最高实现 4.09 的平均接受长度和 3.26 倍加速，同时保持生成质量。代码将在发表后公开。
原文摘要 · Abstract (English)
Speculative Decoding (SD) accelerates Large Language Model (LLM) inference by employing a lightweight draft model to propose candidate tokens, which are verified in parallel by the target model, without compromising generation quality. While Retrieval-based Speculative Decoding (RSD) is favored for its plug-and-play versatility, its potential is impeded by rigid lexical dependencies, rendering both retrieval and verification brittle to surface-level variations. To address this, we propose SENSE (Semantic Embedding Navigation with Soft-gated Evaluation). By anchoring retrieval on the hidden states of the target model, SENSE establishes robust semantic alignment, which empowers the Soft-gated Evaluation module to validate semantic equivalence rather than surface forms. To ensure rigorous benchmarking, we deconstruct existing methods into atomic primitives within a unified framework, facilitating granular, component-level comparison. Extensive experiments across diverse domains demonstrate that SENSE outperforms multiple baselines on the LLaMA and Qwen families, attaining up to 4.09 mean acceptance length and 3.26x speedup, while preserving generation quality. Our code will be released upon publication.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。