PiKV通过分专家存储缓存，降低MoE模型推理的内存开销。
PiKV: KV Cache Management System for Mixture of Experts
- 将专家级缓存分片跨GPU分布，减少全局同步
- 路由与调度机制使缓存访问减少30%以上
- 适合大规模MoE模型部署者使用
随着大语言模型规模和上下文长度持续增长，键值（KV）缓存存储的内存与通信开销已成为多GPU、多节点推理的主要瓶颈。尽管基于MoE架构可稀疏化专家间的计算，但对应的KV缓存仍为稠密且全局同步，造成显著开销。我们提出PiKV，一种专为MoE架构设计的并行分布式缓存服务框架。PiKV采用专家分片的缓存存储方式，在GPU间划分缓存；引入PiKV路由以减少令牌到缓存的访问次数；设计PiKV调度策略自适应保留查询相关条目。为进一步降低内存使用，PiKV在缓存流水线中集成压缩模块以加速处理。PiKV已开源，项目持续迭代，目标是成为面向MoE架构的综合性缓存管理方案。
原文摘要 · Abstract (English)
As large-scale language models continue to scale up in both size and context length, the memory and communication cost of key-value (KV) cache storage has become a major bottleneck in multi-GPU and multi-node inference. While MoE-based architectures sparsify computation across experts, the corresponding KV caches remain dense and globally synchronized, resulting in significant overhead. We introduce \textbf{PiKV}, a parallel and distributed KV cache serving framework tailored for MoE architecture. PiKV leverages \textit{expert-sharded KV storage} to partition caches across GPUs, \textit{PiKV routing} to reduce token-to-KV access, and a \textit{PiKV Scheduling} to adaptively retain query-relevant entries. To further reduce memory usage, PiKV integrates \textit{PiKV Compression} modules the caching pipeline for acceleration. PiKV is recently publicly available as an open-source software library: \href{https://github.com/NoakLiu/PiKV}{https://github.com/NoakLiu/PiKV}. PiKV is still a living project, aiming to become a comprehesive KV Cache management system for MoE Architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。