用并行预测取代逐个生成，让大模型推理快62%。
ParallelSpec: Parallel Drafter for Efficient Speculative Decoding
- 训练并行草稿模型一次预测多个未来词元，摆脱串行依赖。
- 在不同文本任务上延迟降低最高62%，Llama-2-13B整体提速2.84倍。
- 可无缝接入现有方案，训练成本低，适合追求高效推理的场景。
推测式推理已被证明是提升大语言模型（LLM）推理效率的有效方法，其中小型草稿模型以低成本预测未来词元，目标模型并行验证这些预测。然而，大多数现有方法仍采用自回归方式生成草稿词元，以维持语言建模中的序列依赖性，我们认为这在推测式推理中造成了巨大的计算开销。本文提出 ParallelSpec，作为当前先进推测式推理方法中自回归草稿策略的替代方案。与推测阶段的自回归草稿不同，我们训练一个并行草稿模型，以单次推断高效预测多个未来词元。ParallelSpec 可被集成到任何需要对齐草稿模型与目标模型输出分布的推测式推理框架中，且训练成本极低。实验结果表明，ParallelSpec 在多个领域的文本生成基准上将基线方法的延迟最高加速62%，在 Llama-2-13B 模型上使用第三方评估标准实现2.84倍的整体速度提升。
原文摘要 · Abstract (English)
Speculative decoding has proven to be an efficient solution to large language model (LLM) inference, where the small drafter predicts future tokens at a low cost, and the target model is leveraged to verify them in parallel. However, most existing works still draft tokens auto-regressively to maintain sequential dependency in language modeling, which we consider a huge computational burden in speculative decoding. We present ParallelSpec, an alternative to auto-regressive drafting strategies in state-of-the-art speculative decoding approaches. In contrast to auto-regressive drafting in the speculative stage, we train a parallel drafter to serve as an efficient speculative model. ParallelSpec learns to efficiently predict multiple future tokens in parallel using a single model, and it can be integrated into any speculative decoding framework that requires aligning the output distributions of the drafter and the target model with minimal training cost. Experimental results show that ParallelSpec accelerates baseline methods in latency up to 62% on text generation benchmarks from different domains, and it achieves 2.84X overall speedup on the Llama-2-13B model using third-party evaluation criteria.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。