用多层小模型并行预判，加速大模型文本生成。
Fast Inference via Hierarchical Speculative Decoding
- 多层小模型逐级预判，大模型一次性验证
- 实测比单层预判快1.2倍，延迟更低
- 适合追求高速生成的部署场景
Transformer语言模型采用自回归方式生成文本，推理延迟与生成词元数量成正比。推测解码通过使用小型草稿模型预提出词元，由大型目标模型并行验证，从而降低延迟而不损失输出质量。然而实践中存在一系列不同速度与准确率的草稿模型。本文提出分层推测解码（HSD），将多个草稿模型按层级排列，每个模型提出词元，下一更大型模型在单次前向传播中验证，直至最终由目标模型验证。我们推导出任意层级结构的期望延迟表达式，并证明最优层级选择可在多项式时间内完成。实验表明，HSD相比最佳单层草稿基线最多提升1.2倍速度，验证了该算法在降低生成延迟方面的实用性。
原文摘要 · Abstract (English)
Transformer language models generate text autoregressively, making inference latency proportional to the number of tokens generated. Speculative decoding reduces this latency without sacrificing output quality, by leveraging a small draft model to propose tokens that the larger target model verifies in parallel. In practice, however, there may exist a set of potential draft models- ranging from faster but less inaccurate, to slower yet more reliable. We introduce Hierarchical Speculative Decoding (HSD), an algorithm that stacks these draft models into a hierarchy, where each model proposes tokens, and the next larger model verifies them in a single forward pass, until finally the target model verifies tokens. We derive an expression for the expected latency of any such hierarchy and show that selecting the latency-optimal hierarchy can be done in polynomial time. Empirically, HSD gives up to 1.2x speed-up over the best single-draft baseline, demonstrating the practicality of our algorithm in reducing generation latency beyond previous techniques.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。