提出新缓存淘汰策略，让推理模型在压缩4倍内存时更准更快。
Value-Aware Stochastic KV Cache Eviction for Reasoning Models
- 识别大数值状态并保护，避免模型陷入重复推理
- 引入随机性提升缓存多样性，6项任务平均准确率更高
- 无需训练，适配主流框架，适合追求效率的推理部署
推理模型通过长链条思维提升准确性，但其长输出导致内存与计算瓶颈。KV缓存淘汰方法通过移除不重要键值对降低开销，但常导致准确率下降，不如保留完整缓存的稀疏注意力方法。我们发现两个关键因素：一是少数值状态具有异常大的幅值，淘汰它们会导致模型进入重复推理循环；二是淘汰过程引入随机性可提升准确率，增强缓存多样性。基于此，提出无需训练的VaSE策略，保护大幅值值状态并促进多样化淘汰决策。在六项推理任务中，使用VaSE实现4倍缓存压缩的Qwen3模型，平均准确率超过当前最优选择方法，且比最强淘汰方法高出4%以上。该方法弥合了效率与准确性的差距，兼容FlashAttention2，支持推理模型静态内存占用。
原文摘要 · Abstract (English)
Reasoning models improve accuracy through extended chains of thought, but their long outputs create a memory and compute bottleneck. KV cache eviction methods reduce this cost by evicting unimportant key-value pairs from the cache, yet they often yield worse accuracy than selection-based sparse attention alternatives, which keep the full KV cache. We identify key factors crucial to KV cache eviction accuracy. First, a small fraction of value states have abnormally large magnitudes, and evicting them causes catastrophic failure where models enter repetitive reasoning loops. Second, introducing stochasticity during eviction improves accuracy by increasing cache diversity. Based on these findings, we propose Value-aware Stochastic KV Cache Eviction (VaSE), a training-free recipe that protects large-magnitude value states and promotes diverse eviction decisions. Across six reasoning tasks, Qwen3 models using VaSE with 4x KV cache compression yield higher average accuracies than SOTA selection method at the same sparsity, while outperforming the strongest eviction method by more than 4%. Overall, VaSE bridges the gap between efficiency and accuracy, supporting FlashAttention2 and enabling a static memory footprint for reasoning models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。