让被丢弃的草稿变废为宝,提升大模型推理速度3.3倍。
Make Every Draft Count: Hidden State based Speculative Decoding
- 在隐藏状态层面做自回归预测,避免错误标记污染中间结果。
- 通过重用失败草稿生成高质量候选词树,实现推理加速3.3倍。
- 适合追求高效推理、部署大模型的开发者和研究者使用。
推测解码已成为加速大语言模型推理的关键技术,通过轻量级草稿模型生成候选词,由目标模型并行验证。然而该方法导致显著计算浪费:多数草稿词验证失败被丢弃。为回收这部分计算资源,我们提出新系统,将被丢弃的草稿转化为可复用的隐藏状态。核心思路是在隐藏状态层面进行自回归预测,推迟整合词信息,使草稿隐藏状态不受错误词干扰,从而实现复用。首先,设计基于自回归隐藏状态的草稿模型,保留比传统方式更丰富的语义信息;其次,提出高效的词信息注入机制,利用专用草稿模型构建高质量草稿词树,支持从验证失败中重采样;最后,消除设计中的开销以最大化硬件利用率。大量实验表明,相比标准推测解码,最快提速达3.3倍。
原文摘要 · Abstract (English)
Speculative decoding has emerged as a pivotal technique to accelerate LLM inference by employing a lightweight draft model to generate candidate tokens that are subsequently verified by the target model in parallel. However, while this paradigm successfully increases the arithmetic intensity of memory-bound inference, it causes significant compute inefficiency: the majority of draft tokens fail verification and are discarded, resulting in waste of computation. Motivated by the goal of recollecting this wasted computation, we propose a novel system that transforms discarded drafts into reusable tokens. Our key insight is to perform auto-regressive prediction at the hidden states level and postpone the integrating token information after the hidden states generation, so the draft hidden states are not contaminated by incorrect tokens, enabling hidden state reuse. To implement such a system, first we introduce a draft model architecture based on auto-regressive hidden states, which preserves richer semantics than token-based drafters to facilitate draft repurposing. Second, we design an efficient token information injection mechanism that leverages our specialized draft model to construct high-quality draft token trees and enables resampling tokens from verification failures. Third, we eliminate the overhead hidden in our design to further maximize hardware utilization. We conducted extensive evaluations against various baselines, demonstrating up to a 3.3x speedup against standard speculative decoding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。