不改模型结构，让大模型推理快三倍，错误率仅0.37%。
RKSC: Reasoning-Aware KV Cache Sharing and Confident Early Exit for Multi-Step LLM Inference
- 根据语义相似性共享键值缓存，避免重复计算
- 高置信度时跳过验证，中间层熵稳即提前退出
- 动态淘汰缓存，防止无限膨胀，适合部署场景
我们提出RKSC（推理感知键值缓存共享）框架，无需训练即可消除多分支大模型推理中的两项结构性冗余。ASKS通过隐藏层余弦相似度，仅计算一次前缀键值缓存并广播至语义相似分支，严格推广了vLLM和SGLang使用的精确词元前缀缓存。CGEE采用双重退出机制：当生成置信度在分支间足够高时，直接跳过验证前向传播；当逐层熵趋于稳定时，在中间层终止验证过程，仅需对Transformer骨干添加轻量钩子。RSBCM通过注意力加权的深度优先淘汰策略，防止缓存无限增长。在五个模型系列（7B-10B）、四个基准测试及1,000个问题上，RKSC相较无缓存基线平均提速3.008倍（峰值3.990倍），较等效于vLLM的前缀缓存提升1.66倍，且由CGEE引入的错误率为0.37%（1,616次验证调用中仅6次出错）。无需微调或架构修改。代码已开源。
原文摘要 · Abstract (English)
We introduce RKSC (Reasoning-Aware KV Cache Sharing), a training-free inference framework that eliminates two structural redundancies in multi-branch LLM reasoning pipelines. ASKS (Attention-Similarity KV Sharing) computes the prefix KV cache once and broadcasts it to all semantically similar branches via hidden-state cosine similarity, strictly generalising the token-exact prefix caching used by vLLM and SGLang. CGEE (Confidence-Gated Early Exit) applies two complementary exit mechanisms: (1) it skips the verification forward pass entirely when generation confidence is decisive across branches, and (2) it terminates the verification pass at an intermediate layer when per-layer entropy stabilises, using lightweight hooks on the transformer backbone. RSBCM (Reasoning-Selective Block Cache Manager) prevents unbounded cache growth via attention-weighted depth-priority eviction. Across five model families (7B-10B), four benchmarks, and 1,000 evaluated problems, RKSC achieves a mean speedup of 3.008x over the No-KV baseline (peak 3.990x), a 1.66x mean improvement over vLLM-equivalent prefix caching, with a CGEE-induced error rate of only 0.37% (6 errors out of 1,616 verify calls). No fine-tuning or architecture changes are required. Code is available at https://github.com/AnirudhSekar/RKSC.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。