通过分布式缓存热门前缀，显著提升大模型推理速度。
PTStore (Prefix Tensor Store): Distributed Prefix Caching and Replication for High Throughput Inference Serving
- 将常用键值缓存前缀分发复制，避免重复计算。
- 在长文本问答任务上效率比基线高5-6倍。
- 适合需要高速推理的海量请求场景。
受内容分发网络（CDN）客户端缓存设计启发，PTStore将构成可复用键值缓存前缀的热门张量进行分布式存储与复制，这是当前先进方法加速推理的核心技术。该机制降低了访问键值缓存的延迟，并缓解了因部分服务器承载过多热门张量导致的负载不均问题。由于去中心化设计，PTStore使大语言模型推理的键值缓存规模得以扩大数个数量级。结果表明，在长篇问答数据集上，其推理效率比现有基线高出5至6倍，后者无法跨节点和GPU聚合内存，必须重新生成键值缓存。
原文摘要 · Abstract (English)
Inspired by the design of client caching in Content Delivery Networks (CDNs), PTStore distributes and replicates popular tensors that form reusable KV cache prefixes, which are the main technique used by state of art approaches to accelerate inferences. This reduces the latency of accessing the KV cache and alleviates load imbalance caused by a disproportionately large number of requests on servers containing popular tensors. Furthermore, thanks to decentralization, PTStore allows the expansion of the size of the KV cache for LLM inference by orders of magnitude. As a result, PTStore can execute inferences on long passage Q\&A datasets 5-6 times more efficiently than current baselines, which do not aggregate memory across different nodes and GPUs and therefore require regenerating the KV cache.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。