FlowKV通过优化缓存传输与调度，显著降低大模型推理延迟。
FlowKV: A Disaggregated Inference Framework with Low-Latency KV Cache Transfer and Load-Aware Scheduling
- 采用块级调用与连续内存分配，减少缓存传输调用次数。
- KV缓存传输平均延迟从0.944秒降至0.053秒，降幅达96%。
- 支持异构GPU，适合高并发、计算负载不均的场景。
解耦推理已成为提升大语言模型推理吞吐量的重要框架，将预填充（P）和解码（D）阶段分离。然而，P节点与D节点间KV缓存传输存在显著延迟。块级调用方式与不连续的缓存内存分配导致传输内核调用次数增加。现有框架常固定P/D节点角色，造成计算负载失衡。本文提出FlowKV，一种新型解耦推理框架，通过优化缓存传输，将平均传输延迟从0.944秒降低至0.053秒，几乎消除传输时间对总请求延迟的影响。引入负载感知调度器，实现请求均衡调度与灵活的PD节点分配，最大化硬件资源利用率，在正常、计算失衡及极端过载等场景下均达到峰值系统吞吐量。实验表明，相比基线，FlowKV在LongBench数据集上推理加速15.2%-48.9%，并支持异构GPU部署。
原文摘要 · Abstract (English)
Disaggregated inference has become an essential framework that separates the prefill (P) and decode (D) stages in large language model inference to improve throughput. However, the KV cache transfer faces significant delays between prefill and decode nodes. The block-wise calling method and discontinuous KV cache memory allocation increase the number of calls to the transmission kernel. Additionally, existing frameworks often fix the roles of P and D nodes, leading to computational imbalances. In this paper, we propose FlowKV, a novel disaggregated inference framework, which reduces the average transmission latency of KV cache by 96%, from 0.944s to 0.053s, almost eliminating the transfer time relative to the total request latency by optimizing the KV cache transfer. FlowKV introduces the Load-Aware Scheduler for balanced request scheduling and flexible PD node allocation. This design maximizes hardware resource utilization, achieving peak system throughput across various scenarios, including normal, computational imbalance, and extreme overload conditions. Experimental results demonstrate that FlowKV significantly accelerates inference by 15.2%-48.9% on LongBench dataset compared to the baseline and supports applications with heterogeneous GPUs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。