动态调整猜测生成长度，提升大模型推理速度
Token-Driven GammaTune: Adaptive Calibration for Enhanced Speculative Decoding
- 根据令牌接受率自适应调节猜测长度，无需训练
- 平均提速15%~16%，性能波动更小
- 适合需要稳定高速推理的部署场景
推测解码通过小型草稿模型预估令牌，再由大型目标模型验证，以加速大语言模型推理。但选择最优推测长度对最大化加速比同时最小化浪费计算至关重要。我们提出无需训练的自适应算法 GammaTune 与 GammaTune+，基于启发式切换机制，根据令牌接受率动态调整推测长度。在 SpecBench 多任务和模型对上评估，该方法优于其他启发式方法与固定长度推测解码，使用 GammaTune 实现平均 15%（±5%）加速，GammaTune+ 达到 16%（±3%），同时降低性能方差。这使 GammaTune 成为实际部署中稳健高效的解决方案。
原文摘要 · Abstract (English)
Speculative decoding accelerates large language model (LLM) inference by using a smaller draft model to propose tokens, which are then verified by a larger target model. However, selecting an optimal speculation length is critical for maximizing speedup while minimizing wasted computation. We introduce \textit{GammaTune} and \textit{GammaTune+}, training-free adaptive algorithms that dynamically adjust speculation length based on token acceptance rates using a heuristic-based switching mechanism. Evaluated on SpecBench across multiple tasks and model pairs, our method outperforms other heuristic-based approaches and fixed-length speculative decoding, achieving an average speedup of 15\% ($\pm$5\%) with \textit{GammaTune} and 16\% ($\pm$3\%) with \textit{GammaTune+}, while reducing performance variance. This makes \textit{GammaTune} a robust and efficient solution for real-world deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。