通过语法语义一致性提升大模型推理速度，加速文本生成。
S$^4$C: Speculative Sampling with Syntactic and Semantic Coherence for Efficient Inference of Large Language Models
- 多头草稿+连续验证树，实现快速生成与高效校验
- 在Spec-bench上提速2.26至2.60倍，减少资源消耗
- 适合追求高并发、低延迟推理的应用场景
大型语言模型在多种下游任务中展现出强大的推理能力，但其自回归特性导致推理延迟较高，限制了实时应用。推测采样通过引入草稿阶段和并行验证阶段，实现更快的令牌生成与验证。现有方法忽视了文本生成中的内在连贯性，影响效率。为此，我们提出一种结合语法与语义一致性的推测采样框架S$^4$C，利用多头草稿实现快速令牌生成，并采用连续验证树进行高效的候选验证与特征复用。实验表明，S$^4$C在主流任务上优于基线方法，显著提升效率与并行度，能以更少计算资源生成更多有效令牌。在Spec-bench基准测试中，加速比达2.26x-2.60x，超越当前最优方法。
原文摘要 · Abstract (English)
Large language models (LLMs) exhibit remarkable reasoning capabilities across diverse downstream tasks. However, their autoregressive nature leads to substantial inference latency, posing challenges for real-time applications. Speculative sampling mitigates this issue by introducing a drafting phase followed by a parallel validation phase, enabling faster token generation and verification. Existing approaches, however, overlook the inherent coherence in text generation, limiting their efficiency. To address this gap, we propose a Speculative Sampling with Syntactic and Semantic Coherence (S$^4$C) framework, which extends speculative sampling by leveraging multi-head drafting for rapid token generation and a continuous verification tree for efficient candidate validation and feature reuse. Experimental results demonstrate that S$^4$C surpasses baseline methods across mainstream tasks, offering enhanced efficiency, parallelism, and the ability to generate more valid tokens with fewer computational resources. On Spec-bench benchmarks, S$^4$C achieves an acceleration ratio of 2.26x-2.60x, outperforming state-of-the-art methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。