通过并行语义前缀验证,显著提升大模型生成速度。
Parallel Prefix Verification for Speculative Generation

- 用自定义注意力掩码并行验证多个语义前缀,无需逐段检查。
- 相比原模型提速1.25倍至4.3倍,组合使用可达4.5倍。
- 适合追求高吞吐的部署场景,兼容现有加速方法。
我们提出PARSE(PArallel pRefix Speculative Engine),一种通过语义级并行前缀验证加速大语言模型推理的推测生成框架。现有推测解码方法受限于逐标记等价性,导致接受长度短、加速有限。虽语义或分段级验证能提升接受粒度,但先前方法依赖串行验证,引入显著开销。PARSE引入并行前缀验证,在单次前向传播中利用定制注意力掩码评估完整草稿中多个前缀的正确性,直接识别最长有效前缀。该方法消除了串行验证开销,实现高效验证。PARSE与标记级推测解码正交,可组合使用以获额外增益。在多模型与基准测试中,相较目标模型吞吐提升1.25倍至4.3倍,与EAGLE-3组合可达1.6倍至4.5倍,且准确率几乎无损失。这证明并行前缀验证是一种有效通用的LLM推理加速方案。
原文摘要 · Abstract (English)
We introduce PARSE (PArallel pRefix Speculative Engine), a speculative generation framework that accelerates large language model (LLM) inference by parallelizing prefix verification on a semantic level. Existing speculative decoding methods are fundamentally limited by token-level equivalence: the target model must verify each token, leading to short acceptance lengths and modest speedups. Moving to semantic or segment-level verification can substantially increase acceptance granularity, but prior approaches rely on sequential verification, introducing significant overhead and limiting practical gains. PARSE introduces parallel prefix verification, enabling semantic-level verification without sequential checks. Given a full draft from a draft model, the target model evaluates correctness across multiple prefixes in a single forward pass using a custom attention mask, directly identifying the maximal valid prefix. This eliminates sequential segment verification, and makes verification compute-efficient. PARSE is orthogonal to token-level speculative decoding and can be composed with it for additional gains. Across models and benchmarks, PARSE delivers $1.25\times$ to $4.3\times$ throughput gain over the target model, and $1.6\times$ to $4.5\times$ when composed with EAGLE-3, all with negligible accuracy degradation. This demonstrates parallel prefix verification as an effective, general approach to accelerating LLM inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。