通过几何分析优化大模型注意力缓存的比特分配，让关键信息用更高精度存储。
Quantize What Counts: More for Keys, Less for Values
- 基于变换器内在几何特性，发现键矩阵比值矩阵信息密度更高。
- 在相同内存预算下，给键分配更多比特可降低量化误差，准确率提升至98.3%。
- 为高效推理提供理论指导，适合关注模型压缩与部署的开发者。
大型语言模型在推理时面临由注意力键值（KV）缓存引发的内存瓶颈，该缓存随模型规模和上下文长度增长。尽管对KV缓存进行量化可缓解开销，但键与值之间的比特分配常依赖经验调参，缺乏理论依据且泛化性差。本文提出两个定理，将混合精度KV量化建立在Transformer模型的内在几何基础上：首先，键投影的谱范数和Frobenius范数系统性地大于值矩阵，表明键路径具有更高信息密度；其次，在给定内存预算下，优先提高键的精度能严格减少量化误差，并更好保持模型准确率。跨多个主流LLM和基准的实证评估显示，采用键优分配（如4比特键、2比特值）相比均匀分配（如两者均为4比特），可保留高达98.3%的准确率，同时显著节省内存。该成果将比特分配从随意调参转变为理论驱动的几何设计原则。源码见https://github.com/mohsenhariri/spectral-kv。
原文摘要 · Abstract (English)
Large Language Models (LLMs) suffer inference-time memory bottlenecks dominated by the attention Key-Value (KV) cache, which scales with model size and context length. While KV-cache quantization alleviates this cost, bit allocation between keys and values is often tuned heuristically, lacking theoretical grounding and generalizability. This paper proposes two theorems that anchor mixed-precision KV quantization in the intrinsic geometry of Transformer models. First, key projections systematically have larger spectral and Frobenius norms than value matrices, implying higher information density along the key path. Second, for any given memory budget, prioritizing precision for keys over values strictly reduces quantization error and better preserves accuracy. Empirical evaluations across various prominent LLMs and benchmarks show that key-favored allocations (e.g., 4-bit keys, 2-bit values) retain up to 98.3\% accuracy compared to uniform allocations (e.g., 4-bit for both), while conserving memory. These results transform bit allocation from ad hoc tuning into a theoretically grounded, geometry-driven design principle for efficient LLM inference. Source code is available at https://github.com/mohsenhariri/spectral-kv.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。