用两种量化方案配合推测解码，加速大模型推理且不损失精度。
QSpec: Speculative Decoding with Complementary Quantization Schemes
- 用低精度联合量化快速草稿，高精度权重量化精确验证。
- 相比高精度基线提速1.64倍，批量场景下比现有方法快1.55倍。
- 无需重训练，可即插即用，适配不同模型和量化方式。
量化被广泛用于加速大语言模型（LLMs）推理并降低内存占用。尽管激活-权重联合量化能实现高效的低精度推理，但在多步推理任务中性能下降明显。本文提出QSpec，一种新量化范式，通过推测解码将效率与质量解耦：使用低精度联合量化快速生成草稿，高精度权重量化进行精确验证。QSpec在各阶段复用权重和KV缓存，实现近乎零成本切换，无需重训练或辅助模型。相比高精度基线，QSpec在不损失质量的前提下实现最高1.64倍加速；在批量设置下，相比最优推测解码方法提速达1.55倍。此外，QSpec支持即插即用部署，在不同模型规模、量化方法和工作负载间具有良好泛化性。该方案为内存受限场景下的高保真量化大模型服务提供了实用且可扩展的解决方案。代码已开源：https://github.com/hku-netexplo-lab/QSpec。
原文摘要 · Abstract (English)
Quantization is widely adopted to accelerate inference and reduce memory consumption in large language models (LLMs). While activation-weight joint quantization enables efficient low-precision decoding, it suffers from substantial performance degradation on multi-step reasoning tasks. We propose QSpec, a novel quantization paradigm that decouples efficiency from quality by integrating two complementary schemes via speculative decoding: low-precision joint quantization for fast drafting and high-precision weight-only quantization for accurate verification. QSpec reuses both weights and KV cache across stages, enabling near-zero-cost switching without retraining or auxiliary models. Compared to high-precision baselines, QSpec achieves up to 1.64x speedup without quality degradation, and outperforms state-of-the-art speculative decoding methods by up to 1.55x in batched settings. Furthermore, QSpec supports plug-and-play deployment and generalizes well across model scales, quantization methods, and workloads. These properties make QSpec a practical and scalable solution for high-fidelity quantized LLM serving under memory-constrained scenarios. Our code is available at https://github.com/hku-netexplo-lab/QSpec.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。