通过选择性采样与稀疏重建，实现长上下文缓存的高效压缩。
S$^4$R: Selective Sampling, Subspaces, and Sparse Reconstruction for Compressed Long-Context KV Caching
- 从精选令牌构建低秩子空间，减少冗余信息。
- 在解码阶段仅重构关键位置，实现5倍压缩且精度接近全量缓存。
- 无需外部校准数据，适合大规模语言模型长文本生成场景。
大型语言模型（LLMs）上下文窗口的增长显著提升了其长距离依赖建模能力，但带来了巨大的键值（KV）缓存内存开销。尽管低秩压缩是缓解此问题的有前景方案，现有方法面临两难：离线方法依赖外部校准数据，而在线方法需对完整提示进行分解与重建，计算成本高昂。本文提出S$^4$R，通过选择性采样构建低秩子空间，并在解码时采用稀疏重建机制，仅保留重要位置的缓存。该方法利用提示感知初始化，在代表性提示子集上建立初始键值基，平衡了校准数据依赖与预填充开销。由于每次解码步骤完全重建缓存代价过高且影响吞吐量，我们进一步引入稀疏重建策略，仅恢复关键位置。在Llama与Qwen模型家族上针对LongBench和RULER数据集的实验表明，S$^4$R可实现最高5倍的KV缓存压缩，同时保持接近全缓存精度，兼具固定压缩的效率与提示相关方法的适应性。
原文摘要 · Abstract (English)
The growth of context window lengths in Large Language Models (LLMs) significantly enhances their long-context capabilities but incurs prohibitive memory costs due to the Key-Value (KV) cache. Although low-rank compression of KV cache is a promising remedy, existing methods face a dilemma: offline approaches depend on external calibration data, whereas online approaches incur substantial compute for full-prompt decomposition and reconstruction. In this paper, we propose S$^4$R, which builds low-rank subspaces from selectively sampled tokens and computes attention over a sparsely reconstructed KV representation. S$^4$R uses prompt-aware initialization to build initial key/value bases from a representative prompt subset, trading off calibration-data dependence against prefilling cost. Because fully reconstructing the cache at every decoding step is prohibitively expensive and hurts throughput, we further adopt sparse reconstruction to retain only informative positions during decoding. Extensive experiments on LongBench and RULER with Llama and Qwen model families show that S$^4$R achieves up to 5$\times$ KV compression with near full-cache accuracy, combining the efficiency of fixed compression with the adaptability of prompt-dependent methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。