用旋转二值量化加速长文本推理，省显存还更准
RaBitQCache: Rotated Binary Quantization for KVCache in Long Context LLM Inference
- 用随机旋转的二值量化估算注意力权重
- 支持动态调整检索数量，内存读写降低67%
- 适合需要长文本生成的模型部署场景
长上下文大模型推理严重受限于庞大的键值（KV）缓存。现有稀疏注意力方法常因固定预算（Top-k）检索或计算昂贵且有偏差的代理评分而受限。为此，我们提出RaBitQCache，一种新型稀疏注意力框架，利用随机旋转的二值量化与高吞吐二进制INT4运算，高效估算注意力权重。其代理评分是无偏估计器，具有可证明的误差界，支持自适应Top-p检索，能根据实际注意力稀疏度动态调整令牌预算。我们进一步设计了面向硬件的系统，采用异步流水线和延迟更新以掩蔽开销。评估显示，相比最先进基线，RaBitQCache显著加速推理、降低内存输入输出，同时保持生成质量。代码已开源：https://github.com/Sakuraaa0/RaBitQCache.git。
原文摘要 · Abstract (English)
Long-context Large Language Model inference is severely bottlenecked by the massive Key-Value (KV) cache, yet existing sparse attention methods often suffer from static fixed-budget (Top-k) retrieval or rely on proxy scores that are computationally expensive and biased. To address these limitations, we propose RaBitQCache, a novel sparse attention framework that utilizes randomized rotated binary quantization and high-throughput binary-INT4 arithmetic to efficiently estimate attention weights. Our proxy score serves as an unbiased estimator with a proven error bound, enabling adaptive Top-p retrieval that dynamically adjusts the token budget based on actual attention sparsity. We further implement a hardware-aware system with asynchronous pipelining and lazy updates to mask overhead. Evaluations demonstrate that RaBitQCache significantly accelerates inference and reduces memory I/O while preserving generation quality compared to state-of-the-art baselines. Code is available at https://github.com/Sakuraaa0/RaBitQCache.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。