LogQuant用对数机制压缩键值缓存，大幅省显存还提性能。
LogQuant: Log-Distributed 2-Bit Quantization of KV Cache with Superior Accuracy Preservation
- 用对数过滤策略全局压缩KV缓存，不依赖注意力预测。
- 吞吐提升25%，批处理量增60%，数学与代码任务准确率提高40%~200%。
- 可无缝接入transformers，适合大模型推理优化场景。
我们提出LogQuant，一种针对大语言模型推理中键值缓存（KV Cache）的创新2比特量化技术，可在显著节省内存的同时保持优异性能。以往方法或假设后期令牌更重要，或基于早期注意力模式预测关键令牌，但均可能导致性能瓶颈或频繁误判。LogQuant采用基于对数的过滤机制，对整个上下文的KV缓存进行选择性压缩，在相同甚至更低的内存开销下实现更优表现。基准测试显示，其使吞吐提升25%，批处理规模扩大60%，且不增加内存占用；在数学和代码生成等挑战性任务中，相同压缩比下准确率提升40%至200%，优于现有方法。LogQuant可轻松集成至Python的transformers等主流推理框架，代码已开源于https://github.com/Concyclics/LogQuantKV。
原文摘要 · Abstract (English)
We introduce LogQuant, a groundbreaking 2-bit quantization technique for KV Cache in large language model (LLM) inference, delivering substantial memory savings while preserving superior performance. Previous methods either assume that later tokens are more important or attempt to predict important tokens based on earlier attention patterns. Both approaches, however, can result in performance bottlenecks or frequent mispredictions. LogQuant takes a different approach. By applying a log-based filtering mechanism, it selectively compresses the KV Cache across the entire context, achieving better performance with the same or even reduced memory footprint compared to existing methods. In benchmark tests, it enhances throughput by 25% and boosts batch size by 60% without increasing memory consumption. For challenging tasks such as Math and Code Completion, LogQuant improves accuracy by 40% to 200% at the same compression ratio, outperforming comparable techniques.LogQuant integrates effortlessly with popular inference frameworks like Python's transformers library. Implementation can be available in https://github.com/Concyclics/LogQuantKV.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。