把大模型的缓存存到对象存储，降低延迟同时突破内存限制。
ObjectCache: Layerwise Object-Storage Retrieval for KV Cache Reuse
- 将KV缓存存入S3对象存储，按GPU消耗顺序传输以减少等待。
- 64K上下文时延迟仅增加5.6%，4K上下文增加56–75毫秒。
- 适合高并发、长上下文推理场景，降低部署成本。
前缀KV缓存已成为大模型服务的关键机制：通过避免共享前缀（如系统提示）的请求间重复计算，显著降低首次输出时间（TTFT）。然而，累积的KV缓存往往超出GPU内存与本地DRAM容量。为维持低延迟，现有系统将缓存保留在远程DRAM池中，导致服务集群规模和成本上升。本文提出ObjectCache，将KV缓存存储于S3兼容的对象存储中，使容量不再受限，同时最小化对TTFT的影响。ObjectCache协同设计存储协议与传输调度，使存储服务器按GPU消费顺序交付数据，实现数据传输与计算在并发请求间的重叠。我们在100 Gbps RoCE集群上实现了原型，使用NIXL（抽象存储与内存的推理库）、Ceph RGW（集群对象网关）和DAOS（开源存储系统）。对于当前系统常见的64K上下文，ObjectCache仅引入5.6%的延迟；对于4K上下文（计算资源较少以掩盖传输延迟），延迟增加56–75毫秒。在共享带宽受限条件下，其调度器使额外TTFT降低1.2–1.8倍，优于均分带宽方案。
原文摘要 · Abstract (English)
Prefix KV caching has become a key mechanism in LLM serving: it reduces time to first token (TTFT) by avoiding redundant computation across requests that share a prefix (i.e., the system prompt). However, the accumulated KV cache is often larger than what GPU memory and local DRAM can hold. To preserve latency, current systems keep the KV cache in remote DRAM pools, increasing serving-cluster size and cost. In this paper, we explore a different approach: storing the KV cache in S3-compatible object storage so that capacity is no longer the constraint, while minimizing the impact on TTFT. We propose ObjectCache, which co-designs the storage protocol and transfer schedule so that the storage server delivers KV cache data in the order the GPU consumes it, overlapping data transfer with compute across concurrent requests. We prototype ObjectCache on a 100 Gbps RoCE cluster with NIXL (an inference library that abstracts storage and memory), Ceph RGW (an Object Gateway for clusters), and DAOS (an open source storage system). For 64K contexts, common in today's systems, ObjectCache adds only 5.6\% latency over local DRAM; for 4K contexts, where less compute is available to mask transfer, ObjectCache adds 56--75\,ms over the optimal local layerwise baseline. Under shared bandwidth caps, our scheduler reduces added TTFT by 1.2--1.8x compared with equal bandwidth sharing.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。