KVDirect让大模型推理跨节点运行，延迟降55%。
KVDirect: Distributed Disaggregated LLM Inference
- 用张量级通信减少跨节点同步开销。
- 动态调度显卡资源，实现高效缓存传输。
- 拉取式缓存策略降低显卡空闲，适合大规模部署。
大型语言模型（LLMs）已成为众多应用的基础，如风暴般重塑社会。解耦推理（分离prefill与decode阶段）是提升硬件利用率和服务质量的有前景方法。然而，现有系统因节点间通信效率低下，仅支持单机解耦推理，限制了资源灵活分配并降低了服务能力。本文提出KVDirect，通过优化键值缓存（KV cache）传输，实现分布式解耦的LLM推理。主要贡献包括：1）提出新型张量中心通信机制，降低传统分布式GPU系统的同步开销；2）设计专用通信库，支持动态GPU资源调度与高效KV缓存传输；3）引入基于拉取的缓存传输策略，减少GPU空闲时间，提升延迟表现；4）实现开源的LLM推理框架。评估表明，在相同资源约束下，不同工作负载中KVDirect相比基线将每请求延迟降低55%。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have become the new foundation for many applications, reshaping human society like a storm. Disaggregated inference, which separates prefill and decode stages, is a promising approach to improving hardware utilization and service quality. However, due to inefficient inter-node communication, existing systems restrict disaggregated inference to a single node, limiting resource allocation flexibility and reducing service capacity. This paper introduces KVDirect, which optimizes KV cache transfer to enable a distributed disaggregated LLM inference. KVDirect achieves this through the following contributions. First, we propose a novel tensor-centric communication mechanism that reduces the synchronization overhead in traditional distributed GPU systems. Second, we design a custom communication library to support dynamic GPU resource scheduling and efficient KV cache transfer. Third, we introduce a pull-based KV cache transfer strategy that reduces GPU resource idling and improves latency. Finally, we implement KVDirect as an open-source LLM inference framework. Our evaluation demonstrates that KVDirect reduces per-request latency by 55% compared to the baseline across diverse workloads under the same resource constraints.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。