arXiv:2605.02888cs.LGcs.AI2026-05

动态调整大模型推理的猜测长度,提升生成速度。

SpecKV: Adaptive Speculative Decoding with Compression-Aware Gamma Selection

论文配图:SpecKV: Adaptive Speculative Decoding with Compression-Aware Gamma Selection
图 1 · 摘自论文原文
  • 根据草稿模型的置信度和熵值动态选择猜测步长
  • 在不同压缩级别下实现56%的加速,显著优于固定步长
  • 仅增加0.34毫秒开销,适合实际部署

推测解码通过小规模草稿模型预先提出候选词,由大模型验证来加速大语言模型推理。关键超参数γ决定了每步草稿模型提出的词数。现有系统普遍采用固定γ(通常为4),但实证表明最优γ随任务类型变化,且受目标模型压缩程度影响。本文提出SpecKV,一种轻量级自适应控制器,利用草稿模型自身信号每步动态选择γ。我们在4类任务、4种猜测长度、3种压缩级别(FP16、INT8、NF4)下进行全量分析,收集5,112条步级记录,包括每步接受率、草稿熵和草稿置信度。结果表明,最优γ随压缩策略变化,且草稿置信度与熵是接受率的良好预测指标(相关性≈0.56)。SpecKV使用小型MLP基于这些信号优化每轮预期产出词数,在仅增加0.34毫秒决策开销(<0.5%步时)的前提下,相比固定γ=4基线提升56.0%,统计显著(p < 0.001,配对置换检验)。所有采样数据、训练模型与代码笔记均已开源。

原文摘要 · Abstract (English)

Speculative decoding accelerates large language model (LLM) inference by using a small draft model to propose candidate tokens that a larger target model verifies. A critical hyperparameter in this process is the speculation length $γ$, which determines how many tokens the draft model proposes per step. Nearly all existing systems use a fixed $γ$ (typically 4), yet empirical evidence suggests that the optimal value varies across task types and, crucially, depends on the compression level applied to the target model. In this paper, we present SpecKV, a lightweight adaptive controller that selects $γ$ per speculation step using signals extracted from the draft model itself. We profile speculative decoding across 4 task categories, 4 speculation lengths, and 3 compression levels (FP16, INT8, NF4), collecting 5,112 step-level records with per-step acceptance rates, draft entropy, and draft confidence. We demonstrate that the optimal $γ$ shifts across compression regimes and that draft model confidence and entropy are strong predictors of acceptance rate (correlation $\approx$ 0.56). SpecKV uses a small MLP trained on these signals to maximize expected tokens per speculation step, achieving a 56.0% improvement over the fixed-$γ=4$ baseline with only 0.34 ms overhead per decision (<0.5% of step time). The improvement is statistically significant (p < 0.001, paired bootstrap test). We release all profiling data, trained models, and notebooks as open-source artifacts.

推理加速推测解码自适应控制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。