通过预测缓存草案的采纳长度，智能选择验证方式，显著提升大模型生成速度。
Hybrid Verified Decoding: Learning to Allocate Verification in Speculative Decoding
- 用预测代替盲目验证，动态选择用缓存还是模型生成草案。
- 在16个数据集上平均提速2.73倍，尤其适合智能任务场景。
- 揭示了高回报草案集中在小部分生成空间，为运行时选稿提供新思路。
大语言模型生成仍因自回归解码逐字调用模型而成本高昂。推测性解码通过一次性草拟多个词元并验证，降低开销，但加速效果取决于被接受的草稿数量。无参数草稿源可在结构化和智能工作负载中低成本提出长序列，然而某一时刻看似有利的缓存匹配，在后续步骤中可能回报很低。我们提出混合验证解码（Hybrid Verified Decoding），在验证前预测缓存草稿的接受长度，并据此决定采用缓存验证或基于模型的草稿器。在三个大模型和十六个数据集上测试，该方法在智能工作流中表现尤为突出，所有设置下均优于EAGLE3，平均提速2.73倍。分析显示，提示结构创造了缓存机会，高回报草稿集中于草稿空间的一小部分，且基于回报的选稿机制减少了串行解码工作量，表明运行时草稿选择是推测性解码的有前景方向。
原文摘要 · Abstract (English)
Large Language Model (LLM) generation remains expensive because autoregressive decoding calls the model once for each new token. Speculative decoding reduces this cost by drafting multiple tokens and verifying them with the target model in one step, but its speedup depends on how many drafted tokens are accepted. Parameter-free draft sources can propose long continuations at low cost in structured and agentic workloads, yet a cache match that looks promising at one generation step may have low payoff at the next. We propose Hybrid Verified Decoding, which predicts the accepted length of a cache draft before verification and uses this payoff estimate to choose between cache verification and a model-based drafter. Across three LLMs and sixteen datasets, Hybrid Verified Decoding is especially effective on agentic workflows, where it outperforms EAGLE3 in every setting with a 2.73x average speedup. Our analysis shows how prompt structure creates cache opportunities, how high-payoff cache drafts concentrate in a small part of the draft space, and how payoff-guided selection reduces sequential decoding work, pointing to runtime draft selection as a promising direction for speculative decoding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。