XQuant实现亚1.4比特的KV缓存量化，显著降低内存占用。
XQuant: Achieving Ultra-Low Bit KV Cache Quantization with Cross-Layer Compression
- 无需训练，通过跨层压缩和免数据校准实现
- 在长文本任务中实现低于1.4比特的量化，性能优于现有方法
- 适合资源受限环境下的大模型部署，尤其关注内存优化
大语言模型在自然语言处理任务中展现出强大能力，但其在长文本理解与生成过程中产生的KV缓存快速增长，带来显著的内存压力，制约了在资源受限环境中的部署。量化已成为缓解内存消耗、同时保留历史信息的有前景方案。本文提出XQuant，一种无需训练、即插即用的框架，可实现超低等效位宽的KV缓存量化。XQuant引入两项关键创新：计算开销极小的免数据校准方法和跨层KV缓存压缩机制，使量化位宽突破至1.4比特以下。在TruthfulQA与LongBench上的大量实验表明，XQuant在保持更优性能的同时，实现了比当前最优方法（如KIVI-2bit和AsymKV-1.5bit）更低的位宽，建立了内存效率与模型准确率之间的更好权衡。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse natural language processing tasks. However, their extensive memory requirements, particularly due to KV cache growth during long-text understanding and generation, present significant challenges for deployment in resource-constrained environments. Quantization has emerged as a promising solution to reduce memory consumption while preserving historical information. We propose XQuant, a training-free and plug-and-play framework that achieves ultra-low equivalent bit-width KV cache quantization. XQuant introduces two key innovations: a computationally negligible data-free calibration method and cross-layer KV cache compression, enabling quantization to sub-1.4 bits. Extensive experiments on TruthfulQA and LongBench demonstrate that XQuant outperforms state-of-the-art methods (e.g., KIVI-2bit and AsymKV-1.5bit) by achieving lower bit-width while maintaining superior performance, establishing a better trade-off between memory efficiency and model accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。