用低比特量化加速大模型推理中的验证阶段,提升吞吐量。
Quasar: Quantized Self-Speculative Acceleration for Rapid Inference via Memory-Efficient Verification
- 验证阶段采用低比特量化,减少内存占用
- 在保持验证精度前提下,内存流量降低50%
- 无需训练,可与现有草稿策略通用,适合部署加速
推测解码(Speculative Decoding, SD)通过将令牌生成分为快速草稿和并行验证两个阶段,成为加速大语言模型(LLM)推理的主流方法。尽管自推测和前瞻解码等进展已显著降低草稿开销,但验证阶段因需完整前向传播,仍受内存带宽限制,成为主要性能瓶颈。本文提出 extbf{Quasar}(Quantized Self-Speculative Acceleration for Rapid Inference),一种无需训练的新型框架,通过在验证阶段引入低比特量化,突破这一‘内存墙’。实证分析表明,虽然激进的结构剪枝会严重损害验证精度,但基于量化的验证能以高保真度保留对数概率分布,同时有效降低50%内存流量。在 OpenPangu 与 Qwen3 等先进模型上的大量实验显示,Quasar 在保持与全精度方法相当的推测接受长度的同时,实现了 1.28× 的端到端吞吐量提升。该方法与现有草稿策略正交,为推测执行中的验证环节提供了一条通用且高效的加速路径。代码已开源:https://github.com/Tom-HG/Quasar。
原文摘要 · Abstract (English)
Speculative Decoding (SD) has emerged as a premier technique for accelerating Large Language Model (LLM) inference by decoupling token generation into rapid drafting and parallel verification. While recent advancements in self-speculation and lookahead decoding have successfully minimized drafting overhead, they have shifted the primary performance bottleneck to the verification phase. Since verification requires a full forward pass of the target model, it remains strictly memory-bandwidth bound, fundamentally limiting the maximum achievable speedup.In this paper, we introduce \textbf{Quasar} (\textbf{Qua}ntized \textbf{S}elf-speculative \textbf{A}cceleration for \textbf{R}apid Inference), a novel, training-free framework designed to overcome this "memory wall" by employing low-bit quantization specifically for the verification stage. Our empirical analysis reveals that while aggressive structural pruning significantly degrades verification accuracy, quantization-based verification preserves the logit distribution with high fidelity while effectively halving memory traffic. Extensive experiments on state-of-the-art models (e.g., OpenPangu and Qwen3) demonstrate that Quasar maintains a speculative acceptance length comparable to full-precision methods while achieving a $1.28\times$ improvement in end-to-end throughput. Being orthogonal to existing drafting strategies, Quasar offers a generic and efficient pathway to accelerate the verification leg of speculative execution. Code is available at https://github.com/Tom-HG/Quasar.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。