动态短名单提升大词表模型推理速度,兼顾准确与效率。
DynaSpec: Context-aware Dynamic Speculative Sampling for Large-Vocabulary Language Models
- 根据上下文动态选择候选词簇,避免固定短名单的局限性。
- 在稀有词数据集上实现最高2.23倍吞吐量提升,接近全词表性能。
- 适合需要高速推理且词表超百万的大模型应用者。
推测解码通过小模型预生成多个词元,由大模型每步验证一次来加速大语言模型推理。当词表规模超过10⁵时,验证成本基本不变,但小模型的输出投影(O(|V|d))成为瓶颈。现有方法(如FR-Spec、VocabTrim)通过固定频率排序的短名单限制生成,但依赖语料且抑制罕见或领域特定词元,降低接受率并限制加速效果。本文提出DynaSpec,一种面向大词表推测解码的上下文感知动态短名单机制。DynaSpec训练轻量级元分类器,将每个上下文路由至一组粗粒度词元簇;被选中的簇的并集构成小模型的候选集,而大模型仍对全词表进行验证,保证准确性。系统层面,路由与生成计算通过并行流重叠执行,减少端到端开销。在标准推测解码基准测试中,DynaSpec持续提升平均接受长度,对Llama-3-8B达到98.4%的全词表性能,优于固定短名单的93.6%;在含稀有词的数据集上,相比静态方法1.91倍,最高实现2.23倍吞吐量提升。
原文摘要 · Abstract (English)
Speculative decoding accelerates LLM inference by letting a small drafter propose multiple tokens which a large target model verifies once per speculation step. As vocabularies scale past 10e5 tokens,verification cost in the target model is largely unchanged, but the drafter can become bottlenecked by its O(|V|d) output projection. Recent approaches (e.g., FR-Spec, VocabTrim) mitigate this by restricting drafting to a fixed, frequency-ranked shortlist; however, such static truncation is corpus-dependent and suppresses rare or domain-specific tokens, reducing acceptance and limiting speedups. We propose DynaSpec, a context-dependent dynamic shortlisting mechanism for large-vocabulary speculative decoding. DynaSpec trains lightweight meta-classifiers that route each context to a small set of coarse token clusters; the union of the top-selected clusters defines the drafter's shortlist, while the target model still verifies over the full vocabulary, preserving exactness. Systems-wise, routing is overlapped with draft computation via parallel execution streams, reducing end-to-end overhead. Across standard speculative decoding benchmarks, DynaSpec consistently improves mean accepted length-recovering 98.4% of full-vocabulary performance for Llama-3-8B versus 93.6% for fixed-shortlist baselines-and achieves up to a 2.23x throughput gain compared to 1.91x for static approaches on the dataset with rare tokens.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。