用智能网卡实现无干扰的长上下文缓存加速，提升大模型服务效率。
ShadowServe: Interference-Free KV Cache Fetching for Distributed Prefix Caching
- 将缓存数据平面完全卸载到智能网卡，避免主机计算干扰
- 在低带宽下降低1.38倍首令牌延迟，最高提升1.35倍吞吐量
- 适用于需要高并发、低延迟的大模型推理场景
分布式前缀缓存通过复用常见上下文前缀的键值缓存来加速长上下文大模型服务。然而，在网络带宽受限时，缓存获取可能成为瓶颈。压缩可缓解带宽问题，但解压过程可能干扰模型计算，导致性能下降。我们提出ShadowServe，首个基于智能网卡的无干扰前缀缓存系统。ShadowServe将控制平面保留在主机，数据平面完全卸载至智能网卡，消除对主机GPU和CPU的干扰。为克服智能网卡算力与内存资源有限的问题，设计了分块流水线以并行化其计算资源，并采用最小复制内存管理方案降低内存压力。相比现有最优方案，ShadowServe在低带宽场景（≤20 Gbps）下，最高降低1.38倍首令牌延迟（TTFT），加载时间每输出令牌（TPOT）降低2.2倍，吞吐量最高提升1.35倍。
原文摘要 · Abstract (English)
Distributed prefix caching accelerates long-context LLM serving by reusing KV cache entries for common context prefixes. However, KV cache fetches can become a bottleneck when network bandwidth is limited. Compression mitigates the bandwidth issue, but can degrade overall performance when decompression interferes with model computation. We present ShadowServe, the first SmartNIC-accelerated, interference-free prefix caching system for LLM serving. ShadowServe separates a control plane on the host and a data plane fully offloaded to the SmartNIC, which eliminates interference to both host GPU and CPU. To overcome the SmartNIC's limited compute and memory resources, we design a chunked pipeline that parallelizes data plane operations across the SmartNIC's compute resources, and a minimal-copy memory management scheme that reduces memory pressure on the SmartNIC. Compared to state-of-the-art solutions, ShadowServe achieves up to 2.2x lower loaded time-per-output-token (TPOT), and reduces time-to-first-token (TTFT) by up to 1.38x in low-bandwidth scenarios (<= 20 Gbps), translating to up to 1.35x higher throughput.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。