通过智能跳过冗余推理步骤,显著提升大模型推理效率与准确率。
SkipKV: Selective Skipping of KV Generation and Storage for Efficient Inference with Large Reasoning Models
- 基于句子级相似度筛选,选择性删除重复推理内容。
- 相比基线方法,准确率提升26.7%,生成长度缩短40%以上。
- 无需训练,适合追求高效推理的工业部署场景。
大型推理模型(LRMs)因链式思维(CoT)推理过程冗长,导致键值(KV)缓存呈线性增长,带来显著内存开销和吞吐瓶颈。现有KV缓存淘汰方法在多批次场景下表现不佳,因标记级评分不稳定及填充导致有效缓存预算减少,且语义无关的逐标记淘汰会引发重复验证,反而生成更长序列。为此,我们提出SkipKV——一种无需训练的粗粒度、句级选择性淘汰与生成压缩方法。它引入句子评分机制,识别并移除高度相似的句子,保持语义连贯性;同时动态调整控制向量,修正隐藏状态,抑制冗余生成。在多个推理基准测试中,SkipKV在相近压缩预算下,准确率较基线最高提升26.7%;相比最先进方法,生成长度缩短至1.6倍以内,吞吐提升达1.7倍。代码已开源。
原文摘要 · Abstract (English)
Large reasoning models (LRMs) often incur significant key-value (KV) cache overhead, due to their linear growth with the verbose chain-of-thought (CoT) reasoning. This incurs both memory overhead and throughput bottlenecks, limiting efficient deployment. To reduce KV cache size during inference, we first investigate the effectiveness of existing KV cache eviction methods for CoT reasoning. Interestingly, we find that due to unstable token-wise scoring and reduced effective KV budget caused by padding, state-of-the-art (SoTA) eviction methods fail to maintain accuracy in multi-batch settings. Additionally, these methods often generate longer sequences than the original model without eviction, as semantic-unaware token-wise eviction leads to repeated revalidation during reasoning. To address these issues, we present \textbf{SkipKV}, a \textbf{\textit{training-free}} KV compression method that performs selective \textit{eviction} and \textit{generation}, operating at a coarse-grained, sentence-level sequence removal for efficient CoT reasoning. In specific, it introduces a \textit{sentence-scoring metric} to identify and remove highly similar sentences while maintaining semantic coherence. To suppress redundant generation, SkipKV dynamically adjusts a steering vector to update the hidden activation states during inference, enforcing the LRM to generate concise responses. Extensive evaluations on multiple reasoning benchmarks demonstrate that SkipKV achieves up to $\mathbf{26.7}\%$ higher accuracy compared to baseline methods, at a similar compression budget. Additionally, compared to SoTA, SkipKV yields up to $\mathbf{1.6}\times$ shorter generation length while improving throughput by up to $\mathbf{1.7}\times$. Our code is released at: \href{https://github.com/TTTTTTris/SkipKV}{https://github.com/TTTTTTris/SkipKV}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。