用量化小模型多级推测，让大模型推理快2.7倍
ML-SpecQD: Multi-Level Speculative Decoding with Quantized Drafts
- 用MXFP4量化版小模型做推测，无需训练即可直接使用
- 多级推测加速生成过程，最高提速2.72倍
- 适合追求推理速度、不想调参的部署场景
推测解码（SD）是一种在不损失精度的前提下加速大语言模型推理的方法。传统方法需预训练并对齐小模型与大模型，难以即插即用。本文提出使用MXFP4量化的小模型作为推测器，仅通过将大模型权重直接转换为MXFP4即可实现，无需额外训练。该方案在实际中可带来高达2倍的加速。进一步地，我们利用更小的模型对MXFP4模型的生成过程进行二级推测，形成多级推测架构——ML-SpecQD。结合量化与多级推测，相比现有最优方案，最大加速达2.72倍，且保持全精度性能。
原文摘要 · Abstract (English)
Speculative decoding (SD) has emerged as a method to accelerate LLM inference without sacrificing any accuracy over the 16-bit model inference. In a typical SD setup, the idea is to use a full-precision, small, fast model as "draft" to generate the next few tokens and use the "target" large model to verify the draft-generated tokens. The efficacy of this method heavily relies on the acceptance ratio of the draft-generated tokens and the relative token throughput of the draft versus the target model. Nevertheless, an efficient SD pipeline requires pre-training and aligning the draft model to the target model, making it impractical for LLM inference in a plug-and-play fashion. In this work, we propose using MXFP4 models as drafts in a plug-and-play fashion since the MXFP4 Weight-Only-Quantization (WOQ) merely direct-casts the BF16 target model weights to MXFP4. In practice, our plug-and-play solution gives speedups up to 2x over the BF16 baseline. Then we pursue an opportunity for further acceleration: the MXFP4 draft token generation itself can be accelerated via speculative decoding by using yet another smaller draft. We call our method ML-SpecQD: Multi-Level Speculative Decoding with Quantized Drafts since it recursively applies speculation for accelerating the draft-token generation. Combining Multi-Level Speculative Decoding with MXFP4 Quantized Drafts we outperform state-of-the-art speculative decoding, yielding speedups up to 2.72x over the BF16 baseline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。