动态调整生成长度,让大模型推理更稳更快。
DSDE: Dynamic Speculative Decoding with KLD Stability for Real-World Serving
- 用KLD方差判断生成稳定性,动态调节推测长度。
- 在多样请求下延迟媲美顶尖方法,低通过率场景仍稳定。
- 无需训练,适合真实服务场景的智能加速系统。
推测解码可加速大语言模型推理,但在多请求批量服务中固定推测长度效果不佳。本文提出无需训练的动态推测解码引擎DSDE,基于两个核心组件:(1) 利用KL散度方差构建预测信号,诊断生成过程的区域稳定性;(2) 动态调整推测长度上限,缓解序列级延迟尾部问题。实验表明,基于KLD稳定性信号的动态适应具有潜力。该算法在端到端延迟上达到领先基线水平,并在多样化负载下表现更稳健,尤其在低接受率场景仍保持诊断有效性。这些发现验证了后验信号对构建更鲁棒、智能的LLM推理系统的价值,为动态推测长度适配提供了新方向。
原文摘要 · Abstract (English)
Speculative decoding accelerates large language model inference, but its reliance on a fixed speculation length is suboptimal in large-batch serving environments with diverse requests. This paper explores a new direction for dynamic adaptation by investigating a novel class of post-hoc, diagnostic signals. We propose Dynamic Speculative Decoding Engine (DSDE), a training-free framework built on two primary components: (1) a predictive signal based on the variance of the Kullback-Leibler (KLD) divergence, which diagnoses the generation's regional stability, and (2) an adaptive speculation length cap to mitigate the straggler problem in per-sequence decoding. Experiments demonstrate the potential of using KLD-based stability signals for dynamic adaptation. An algorithm guided by these signals achieves end-to-end latency competitive with leading baselines and exhibits superior robustness across diverse workloads. This robustness is particularly valuable in challenging low-acceptance-rate regimes, where the proposed signal maintains its diagnostic utility. Collectively, these findings validate post-hoc signals as a valuable component for building more robust and intelligent LLM inference systems, and highlight a promising direction for future research on dynamic speculation length adaptation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。