用分层4比特量化加速长文本推理，提升速度与内存效率
QuantSpec: Self-Speculative Decoding with Hierarchical Quantized KV Cache
- 自洽推测解码框架，共享目标模型结构但用4比特量化键值缓存和权重
- 在长上下文场景下实现超90%接受率，端到端提速近2.5倍
- 适合边缘设备部署的高效长文本生成任务
大型语言模型（LLMs）正越来越多地部署在边缘设备上处理长上下文任务，对快速高效的长上下文推理需求日益增长。在此类场景中，键值（KV）缓存是显存和延迟的主要瓶颈，因为每个解码步骤都需加载完整缓存。尽管推测解码是广泛采用的加速自回归解码技术，但现有方法因缓存优化策略低效，常导致接受率低，难以获得显著加速。为此，我们提出一种新型自洽推测解码框架 QuantSpec：其草稿模型与目标模型共享架构，但采用分层4比特量化KV缓存和4比特量化权重以实现加速。QuantSpec保持超过90%的接受率，稳定提供高达约2.5倍的端到端加速，优于使用稀疏KV缓存的其他自洽推测方法；同时相较这些方案，内存占用减少约1.3倍。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are increasingly being deployed on edge devices for long-context settings, creating a growing need for fast and efficient long-context inference. In these scenarios, the Key-Value (KV) cache is the primary bottleneck in terms of both GPU memory and latency, as the full KV cache must be loaded for each decoding step. While speculative decoding is a widely accepted technique to accelerate autoregressive decoding, existing methods often struggle to achieve significant speedups due to inefficient KV cache optimization strategies and result in low acceptance rates. To address these challenges, we propose a novel self-speculative decoding framework, QuantSpec, where the draft model shares the architecture of the target model but employs a hierarchical 4-bit quantized KV cache and 4-bit quantized weights for acceleration. QuantSpec maintains high acceptance rates ($>$90%) and reliably provides consistent end-to-end speedups upto $\sim2.5\times$, outperforming other self-speculative decoding methods that use sparse KV cache for long-context LLM inference. QuantSpec also reduces the memory requirements by $\sim 1.3\times$ compared to these alternatives.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。