提出StructKV,通过识别全局信息枢纽提升长文本推理效率。
StructKV: Preserving the Structural Skeleton for Scalable Long-Context Inference

- 用跨层注意力聚合识别全局关键信息节点
- 动态定位最优压缩层,减少关键信息丢失
- 分离计算与存储预算,支持百万级上下文
随着大语言模型上下文窗口突破百万词元,键值缓存的线性增长带来严重的内存容量与带宽瓶颈,制约长上下文推理效率。现有压缩方法通常依赖特定层的局部显著性指标进行令牌筛选,但会系统性丢弃在多层网络中充当全局信息枢纽的临时静默令牌。为此,我们提出StructKV,一种结构感知的键值缓存压缩框架,包含三项核心创新:首先,全局入度中心性通过跨网络深度聚合注意力模式,识别全局信息枢纽;其次,动态枢轴检测利用信息论度量自适应定位最优压缩层;最后,结构传播与解耦将计算预算与存储预算分离。在LongBench和RULER基准上的实验表明,StructKV能有效保留长程依赖与检索鲁棒性。
原文摘要 · Abstract (English)
As Large Language Models (LLMs) scale to support context windows exceeding one million tokens, the linear growth of Key-Value (KV) cache imposes severe memory capacity and bandwidth bottlenecks, constraining the efficiency of long-context inference. Existing compression approaches typically prioritize tokens based on local saliency metrics to decouple prefill computation from decoding memory. However, these methods often rely on local saliency snapshots at a specific layer, thereby systematically discarding tokens that act as global information hubs across the network depth but appear temporarily dormant at the specific layer selected for pruning. To address this limitation, we propose StructKV, a structure-aware KV cache compression framework that introduces three core innovations: First, Global In-Degree Centrality aggregates attention patterns across the network depth to identify global information hubs. Second, Dynamic Pivot Detection utilizes information-theoretic metrics to adaptively locate the optimal layer for compression. Finally, Structural Propagation and Decoupling separates the computational budget from the memory storage budget. Experimental results on the LongBench and RULER benchmarks demonstrate that StructKV effectively preserves long-range dependencies and retrieval robustness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。