让大模型生成更快：异步多设备并行推测解码
AMUSD: Asynchronous Multi-Device Speculative Decoding for LLM Acceleration
- 用小模型快速出候选，大模型并行验证，两模型分设备异步运行
- 平均比推测解码快29%，最高达1.96倍，输出质量不变
- 适合追求推理加速的部署者，开源可直接使用
大语言模型通常采用自回归方式逐个生成标记，每个标记作为下一个的输入。近期推测解码方法通过使用更小、更快的草稿模型快速生成候选标记，再由更大的验证模型并行验证，相比单独使用大模型自回归生成有整体加速效果。本文提出AMUSD（异步多设备推测解码），将草稿与验证阶段解耦为持续异步流程。不同于传统推测解码中仅一个模型（草稿或验证）在某一时刻生成标记，AMUSD使两个模型可在不同设备（如GPU）上独立预测。我们在多个数据集上评估该方法，结果表明，相比推测解码平均提升29%，最高达1.96倍速度提升，且输出质量完全一致。系统已开源，地址见https://github.com/BradMcDanel/AMUSD/
原文摘要 · Abstract (English)
Large language models typically generate tokens autoregressively, using each token as input for the next. Recent work on Speculative Decoding has sought to accelerate this process by employing a smaller, faster draft model to more quickly generate candidate tokens. These candidates are then verified in parallel by the larger (original) verify model, resulting in overall speedup compared to using the larger model by itself in an autoregressive fashion. In this work, we introduce AMUSD (Asynchronous Multi-device Speculative Decoding), a system that further accelerates generation by decoupling the draft and verify phases into a continuous, asynchronous approach. Unlike conventional speculative decoding, where only one model (draft or verify) performs token generation at a time, AMUSD enables both models to perform predictions independently on separate devices (e.g., GPUs). We evaluate our approach over multiple datasets and show that AMUSD achieves an average 29% improvement over speculative decoding and up to 1.96$\times$ speedup over conventional autoregressive decoding, while achieving identical output quality. Our system is open-source and available at https://github.com/BradMcDanel/AMUSD/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。