通过频域压缩键值缓存，实现长文本上下文高效处理
FreqKV: Key-Value Compression in Frequency Domain for Context Window Extension
- 在频域迭代压缩键值缓存，不依赖训练和模型架构
- 将LLaMA-2-7B上下文扩展至256K tokens，困惑度稳定
- 适合需要长文本理解与生成的场景，如文档分析
现有大语言模型的键值缓存压缩方法多依赖令牌淘汰，易丢失关键局部信息，尤其在超出预训练上下文长度时性能急剧下降。基于频域中上下文信息集中于低频成分的观察，我们提出无需参数、与架构无关的FreqKV方法。该方法在频域中迭代压缩不断增长的键值缓存，使模型能高效处理长序列。仅需在8K长度下进行少量训练，FreqKV即可将LLaMA-2-7B的上下文窗口扩展至256K tokens，并保持稳定的困惑度。在预填充与解码任务上的大量实验表明，FreqKV在长上下文理解与生成任务中显著优于现有方法，在LLaMA-2与LLaMA-3上均表现更优。
原文摘要 · Abstract (English)
Existing key-value (KV) cache compression methods for large language models (LLMs) often rely on token eviction, which risks losing critical local information in both long prefilling and decoding scenarios. When extrapolating beyond the pretrained context length, their performance degrades sharply on long-context benchmarks. Motivated by the observation in the frequency domain that the context information is concentrated in the low-frequency components, we propose FreqKV, a parameter-free and architecture-agnostic approach. It iteratively compresses the increasing KV cache in the frequency domain, allowing models to process lengthy contexts efficiently. With minimal training at 8K length, FreqKV extends the context window of LLaMA-2-7B up to 256K tokens while maintaining stable perplexity. Extensive experiments across prefilling and decoding demonstrate that FreqKV enables robust context window extension and consistently outperforms existing KV cache compression methods on LLaMA-2 and LLaMA-3, highlighting its effectiveness for both understanding and generation in long contexts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。