通过总结能力识别关键注意力头，实现大模型长文本推理的内存压缩
SGD-KV: Summarization Guided KV Cache Compression
- 基于摘要任务识别不同注意力头的层级信息聚合能力
- 在100万词上下文中将内存占用减少75%且保持最优性能
- 适合需要高效长文本生成的部署场景
大型语言模型在长上下文推理中面临严重的内存瓶颈，主要源于键值（KV）缓存随上下文长度线性增长。现有压缩方法多依赖简单启发式策略，忽视了不同注意力头的功能差异。本文提出SGD-KV（摘要引导的KV缓存压缩），一种头感知框架，利用新颖的块摘要诊断任务系统识别并优先处理擅长层级信息聚合的注意力头。在Qwen2.5-7B-1M与Qwen3-32B模型上，多个长上下文基准测试表明，SGD-KV在长达100万词的上下文中达到当前最佳性能，同时将KV缓存内存使用量最多降低75%。研究发现，根据注意力头的摘要得分分布进行缓存预算的智能分配，可实现长上下文推理中更优的效率-精度权衡。
原文摘要 · Abstract (English)
Large language models (LLMs) face severe memory bottlenecks in long-context inference due to the linearly growing size of key-value (KV) caches. Existing KV cache compression techniques typically rely on simple heuristics, overlooking the distinct functional roles of different attention heads. We present SGD-KV (Summarization-Guided KV Cache Compression), a head-aware framework that leverages a novel chunk-summarization diagnostic task to systematically identify and prioritize attention heads specialized in hierarchical information aggregation. Experiments on Qwen2.5-7B-1M and Qwen3-32B across diverse long-context benchmarks demonstrate that SGD-KV achieves state-of-the-art performance with contexts up to 1M tokens, while reducing KV cache memory usage by up to 75%. Our findings show that strategically allocating the KV cache budget based on the summarization score distribution of attention heads yields a superior efficiency-accuracy trade-off for long-context inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。