研究长文本任务中键值缓存卸载的精度下降问题,提出更可靠的替代方案。
KV Cache Offloading for Context-Intensive Tasks
- 针对长文本信息提取任务,测试键值缓存卸载效果
- 发现主流模型在关键任务上准确率显著下降
- 提出简单有效的新策略,提升多模型跨任务表现
随着长上下文大模型在各类应用中需求增长,键值(KV)缓存已成为延迟和内存使用的瓶颈。近期,KV缓存卸载被视为降低内存占用与推理延迟、同时保持精度的有前景方法。但以往评估主要集中在无需大量提取上下文信息的任务上。本文聚焦于需要从输入提示中检索大量信息的上下文密集型任务,构建并发布了Text2JSON基准,该任务要求从原始文本中提取结构化知识。我们在Text2JSON及其他上下文密集型任务上评估了现代KV卸载技术,发现其在Llama 3和Qwen 3模型上均出现显著性能下降。分析揭示两大主因:键的低秩投影与不可靠的地标点选取,并提出一种更简单的替代策略,在多个大模型家族和基准上显著提升了准确率。这些发现强调了对长上下文压缩技术进行全面、严谨评估的必要性。
原文摘要 · Abstract (English)
With the growing demand for long-context LLMs across a wide range of applications, the key-value (KV) cache has become a critical bottleneck for both latency and memory usage. Recently, KV-cache offloading has emerged as a promising approach to reduce memory footprint and inference latency while preserving accuracy. Prior evaluations have largely focused on tasks that do not require extracting large amounts of information from the context. In this work, we study KV-cache offloading on context-intensive tasks: problems where the solution requires looking up a lot of information from the input prompt. We create and release the Text2JSON benchmark, a highly context-intensive task that requires extracting structured knowledge from raw text. We evaluate modern KV offloading on Text2JSON and other context-intensive tasks and find significant performance degradation on both Llama 3 and Qwen 3 models. Our analysis identifies two key reasons for poor accuracy: low-rank projection of keys and unreliable landmarks, and proposes a simpler alternative strategy that significantly improves accuracy across multiple LLM families and benchmarks. These findings highlight the need for a comprehensive and rigorous evaluation of long-context compression techniques.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。