通过自适应压缩注意力缓存，让大模型推理更快更省显存。
ThinKV: Thought-Adaptive KV Cache Compression for Efficient Reasoning Models
- 根据思维重要性动态分配令牌精度并逐步淘汰低重要性内容。
- 在数学与编程任务上仅用不到5%原缓存大小，精度近乎无损。
- 适合需要长推理链的大模型部署，尤其看重效率的场景。
大型推理模型生成长输出上下文时，会显著增加关键-值（KV）缓存，迅速耗尽GPU内存。为应对该问题，本文提出ThinKV——一种基于思维自适应的KV缓存压缩框架。其核心观察是：注意力稀疏性可揭示不同思维类型的重要性差异。ThinKV采用量化与淘汰相结合的混合策略，按思维重要性分配令牌精度，并随推理进程逐步剔除低重要性思维中的内容。此外，为实现高效管理，设计了一种扩展PagedAttention的内核，支持被淘汰令牌内存槽的重用，彻底消除内存紧缩开销。在DeepSeek-R1-Distill、GPT-OSS和NVIDIA AceReason多个模型上，针对数学与编码基准的实验表明，ThinKV在仅使用原缓存5%以下的情况下，保持近似无损精度，且推理吞吐量相比最先进方法最高提升5.8倍。
原文摘要 · Abstract (English)
The long-output context generation of large reasoning models enables extended chain of thought (CoT) but also drives rapid growth of the key-value (KV) cache, quickly overwhelming GPU memory. To address this challenge, we propose ThinKV, a thought-adaptive KV cache compression framework. ThinKV is based on the observation that attention sparsity reveals distinct thought types with varying importance within the CoT. It applies a hybrid quantization-eviction strategy, assigning token precision by thought importance and progressively evicting tokens from less critical thoughts as reasoning trajectories evolve. Furthermore, to implement ThinKV, we design a kernel that extends PagedAttention to enable efficient reuse of evicted tokens' memory slots, eliminating compaction overheads. Extensive experiments on DeepSeek-R1-Distill, GPT-OSS, and NVIDIA AceReason across mathematics and coding benchmarks show that ThinKV achieves near-lossless accuracy with less than 5% of the original KV cache, while improving performance with up to 5.8x higher inference throughput over state-of-the-art baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。