通过共享相似推理步骤的键值缓存，大幅降低大模型推理延迟。
ReasonCache: Accelerating Large Reasoning Model Serving through KV Cache Sharing
- 利用协同过滤识别可复用的键值缓存块，实现零拷贝共享。
- 峰值吞吐提升89.2%，平均提升40%-60%，延迟显著降低。
- 适合高并发推理服务，尤其在长序列生成场景下优势明显。
大型推理模型（LRMs）正广泛应用于各类AI推理系统，其强大的推理能力提升了系统性能。然而，在生产环境中部署时面临显著的服务质量挑战：长时间自回归推理带来的巨大内存开销严重限制了吞吐量并增加了延迟，影响并发用户的体验。我们观察到，LRMs经常生成高度相似的中间推理步骤，这些步骤对应各层中高度相似的键值（KV）缓存状态。基于这一发现，我们提出ReasonCache，一种新型的KV缓存管理方法，旨在提升AI推理系统的服务质量。ReasonCache采用协同过滤算法高效识别可复用的KV缓存块，并支持零拷贝缓存重用。实验表明，ReasonCache实现了89.2%的峰值吞吐提升，平均提升达40%-60%，同时相比现有技术保持更高准确性，显著增强了推理服务的响应速度与成本效益。
原文摘要 · Abstract (English)
Large Reasoning Models (LRMs) are becoming integral to many AI inference systems, enhancing their capabilities with advanced reasoning. However, deploying these models in production environments presents a significant QoS challenge: the substantial memory overhead from their long, auto-regressive inference processes severely limits throughput and increases latency, thereby affecting the quality of service for concurrent users. We observe that LRMs frequently generate highly similar intermediate reasoning steps, which, in turn, correspond to highly similar KV cache states across layers. Building on this insight, we propose ReasonCache, a novel KV cache management approach designed to improve the QoS of AI inference systems. ReasonCache utilizes a Collaborative Filtering Algorithm to efficiently identify reusable KV cache blocks and enables zero-copy cache reuse. Experimental evaluation demonstrates that ReasonCache achieves a peak throughput improvement of 89.2% and an average gain of 40-60%, leading to more responsive and cost-effective AI inference services. Notably, this performance is achieved while maintaining higher accuracy compared to existing KV cache management techniques.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。