通过分析模型内部状态，实现语义级并行生成，显著提升大模型推理速度。
Beyond Tokens: Semantic-Aware Speculative Decoding for Efficient Inference by Probing Internal States
- 基于模型隐藏状态探测语义概率，验证完整语义序列而非单个词元
- 在DeepSeekR1-32B上最高提速2.7倍，QwQ-32B上提速2.1倍
- 适合需要长思维链推理的大型语言模型部署场景
大语言模型（LLMs）在诸多任务中表现优异，但因自回归解码导致推理延迟高。这一问题在长思维链模型（LRMs）中尤为突出。尽管推测性解码可通过并行起草和验证多个词元加速推理，现有方法仅在词元层面操作，忽略语义等价性（即不同词元序列表达相同含义），造成大量无效拒绝。本文提出一种语义感知的推测性解码框架SemanticSpec，通过探测模型内部隐藏状态，评估生成特定语义序列的概率，从而验证整个语义序列。在四个基准测试中，SemanticSpec在DeepSeekR1-32B上实现最高2.7倍加速，在QwQ-32B上实现2.1倍加速，持续优于词元级与序列级基线方法，在效率与有效性上均表现更优。
原文摘要 · Abstract (English)
Large Language Models (LLMs) achieve strong performance across many tasks but suffer from high inference latency due to autoregressive decoding. The issue is exacerbated in Large Reasoning Models (LRMs), which generate lengthy chains of thought. While speculative decoding accelerates inference by drafting and verifying multiple tokens in parallel, existing methods operate at the token level and ignore semantic equivalence (i.e., different token sequences expressing the same meaning), leading to inefficient rejections. We propose SemanticSpec, a semantic-aware speculative decoding framework that verifies entire semantic sequences instead of tokens. SemanticSpec introduces a semantic probability estimation mechanism that probes the model's internal hidden states to assess the likelihood of generating sequences with specific meanings. Experiments on four benchmarks show that SemanticSpec achieves up to 2.7x speedup on DeepSeekR1-32B and 2.1x on QwQ-32B, consistently outperforming token-level and sequence-level baselines in both efficiency and effectiveness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。