SWAN通过无解压压缩KV缓存，实现低内存推理且性能稳定。
SWAN: Sparse Winnowed Attention for Reduced Inference Memory via Decompression-Free KV-Cache Compression
- 用正交矩阵旋转并剪枝KV缓存，直接用于注意力计算，无需解压
- 在每令牌节省50%-60%内存时，仍保持接近原始模型的性能
- 支持运行时动态调节压缩率，适合长文本服务场景
大型语言模型在自回归推理中因键值（KV）缓存占用巨大内存而面临瓶颈。现有压缩方法如令牌剔除、量化或低秩技术常导致信息丢失、压缩上限固定，或因显式解压步骤引入显著计算开销。本文提出SWAN，一种无需微调的新框架，彻底消除解压开销。该方法利用离线正交矩阵对KV缓存进行旋转与剪枝，并直接用于注意力计算，无需重建。大量实验表明，SWAN结合少量密集缓冲区，在每令牌节省50%-60%内存的情况下，仍能保持接近未压缩基线的性能。其关键优势在于支持运行时可调压缩率，使部署者可根据需求动态调整内存占用，这种灵活性是依赖固定离线配置的方法所不具备的。结合无解压设计、高压缩下的高性能表现及可调性，SWAN为长上下文大模型服务提供了一种实用高效的解决方案。
原文摘要 · Abstract (English)
Large Language Models (LLMs) face a significant bottleneck during autoregressive inference due to the massive memory footprint of the Key-Value (KV) cache. Existing compression techniques like token eviction, quantization, or other low-rank methods often risk information loss, have fixed limits, or introduce significant computational overhead from explicit decompression steps. In this work, we introduce SWAN, a novel, fine-tuning-free framework that eliminates this overhead. Our method uses an offline orthogonal matrix to rotate and prune the KV-cache, which is then used directly in the attention computation without any reconstruction. Our extensive experiments demonstrate that SWAN, augmented with a small dense buffer, offers a robust trade-off, maintaining performance close to the uncompressed baseline even at aggressive 50-60% memory savings per-token on KV-cache. A key advantage is its runtime-tunable compression level, allowing operators to dynamically adjust the memory footprint, a flexibility absent in methods requiring fixed offline configurations. This combination of a decompression-free design, high performance under compression, and adaptability makes SWAN a practical and efficient solution for serving LLMs with long contexts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。