通过分层语义记忆实现长文本推理的高效缓存,大幅降低显存占用。
SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference

- 按语义熵分块存储,结合GPU-CPU分层缓存架构
- 在128K上下文中减少53.3%显存,平均性能提升5.9%
- 支持动态重建关键内容,适合大模型长文本推理场景
大语言模型在长上下文场景下运行时,KV缓存成为主要内存瓶颈:其大小随序列长度线性增长且需全程保留,全量GPU缓存成本过高。现有压缩方法难以兼顾效率与上下文保真度——逐标记淘汰会丢失信息,语义分组则在预填充阶段固定压缩策略,生成时无法恢复被压缩段的粒度细节。为此,我们提出SeKV,一种基于语义的自适应分辨率KV缓存机制,将上下文组织为熵引导的语义段,跨GPU-CPU层级存储。每段在GPU保留轻量摘要向量用于粗粒度路由,在CPU保存低秩SVD基底以按需重建令牌级内容。训练后的‘放大’机制可选择性展开相关段落,实现精准检索而不将完整缓存加载至GPU。SeKV在保持原模型完全冻结的同时,引入少于0.05%可训练参数,四个基准测试中相比最强语义压缩基线平均提升5.9%,在128K上下文下相比全缓存减少53.3%显存占用。代码已开源于https://github.com/AmirAbaskohi/SeKV。
原文摘要 · Abstract (English)
Large language models increasingly operate over long contexts, where the KV cache becomes a dominant memory bottleneck: its size grows linearly with sequence length and must be retained throughout decoding, making full GPU caching prohibitively expensive without compression. Existing KV cache compression methods struggle to balance efficiency with faithful context preservation. Token eviction discards information, while semantic grouping fixes compression decisions at prefill time; neither can recover token-level detail from a compressed span once it becomes relevant during generation. As a solution, we propose SeKV, a resolution-adaptive semantic KV cache that organizes context into entropy-guided semantic spans and stores them across a GPU-CPU memory hierarchy without discarding information. Each span keeps a lightweight summary vector on GPU for coarse routing and a low-rank SVD basis on CPU for on-demand token-level reconstruction. A trained zoom-in mechanism selectively expands query-relevant spans during decoding, enabling precise retrieval without materializing the full KV cache on GPU. SeKV enables adaptive token-level reconstruction while keeping the base LLM fully frozen and adding fewer than 0.05% trainable parameters. Across four benchmarks, SeKV improves over the strongest semantic compression baseline by 5.9% on average while reducing GPU memory by 53.3% versus full KV caching at 128K context. Code is available on https://github.com/AmirAbaskohi/SeKV.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。