提出利用键值缓存异构性，无训练实现长文本高效推理
Homogeneous Keys, Heterogeneous Values: Exploiting Local KV Cache Asymmetry for Long-Context LLMs
- 基于键的局部同质性合并键，值保持异构分布
- 在LLaMA3.1-8B上LongBench得分43.95，超越当前最优方法
- 无需训练，适配多种模型和长文本任务
大型语言模型扩展上下文长度日益重要，但注意力机制的二次复杂度带来显著挑战。键值缓存压缩成为关键解决方案。通过大量实验分析，我们发现了一个此前被忽视的根本性不对称：相邻键具有相似注意力权重（局部同质性），而相邻值则表现出明显异构分布。这种键值不对称揭示了现有压缩方法统一处理键值的局限性。为此，我们提出一种无训练压缩框架AsymKV，结合基于同质性的键合并与数学证明的无损值压缩。大量实验证明，AsymKV在多个任务和基础模型上均优于现有长上下文方法。例如，在LLaMA3.1-8B上，AsymKV在LongBench上的平均得分为43.95，显著优于当前最优方法H$_2$O的38.89。
原文摘要 · Abstract (English)
Recent advances in Large Language Models (LLMs) have highlighted the critical importance of extending context length, yet the quadratic complexity of attention mechanisms poses significant challenges for efficient long-context modeling. KV cache compression has emerged as a key approach to address this challenge. Through extensive empirical analysis, we reveal a fundamental yet previously overlooked asymmetry in KV caches: while adjacent keys receive similar attention weights ({\it local homogeneity}), adjacent values demonstrate distinct {\it heterogeneous} distributions. This key-value asymmetry reveals a critical limitation in existing compression methods that treat keys and values uniformly. To address the limitation, we propose a training-free compression framework (AsymKV) that combines homogeneity-based key merging with a mathematically proven lossless value compression. Extensive experiments demonstrate that AsymKV consistently outperforms existing long-context methods across various tasks and base models. For example, on LLaMA3.1-8B, AsymKV achieves an average score of 43.95 on LongBench, surpassing SOTA methods like H$_2$O (38.89) by a large margin.Our code can be found in this link:https://github.com/the-scale-lab/Asymkv.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。