考虑网络距离与拥堵的调度策略，显著降低大模型推理首字延迟。
NetKV: Network-Aware Decode Instance Selection for Disaggregated LLM Inference
- 引入网络代价探针，动态评估预填充与解码实例间通信成本。
- 在64卡模拟环境中，平均首字延迟降低21.2%，服务等级达标率提升20.1%。
- 无需改硬件或框架，适合大规模分布式大模型部署场景。
解耦式大模型推理将键值缓存置于数据中心网络中传输，导致传输时间直接计入首字时间（TTFT）预算。现有调度器仅依据计算负载和前缀缓存局部性，忽略预填充与解码实例间的拓扑距离与动态拥塞。本文提出轻量级调度接口——网络代价探针，并证明忽略网络因素会使仅基于缓存的调度在上下文长度增长时任意次优。NetKV是一种每请求复杂度为O(|D|)的贪心调度算法，利用该探针，其层级排序对过时遥测具有理论鲁棒性。在基于Mooncake数据集、64张GPU的四层胖树模拟器上，NetKV相比轮询调度将平均TTFT降低21.2%，相较调优后的缓存+负载感知调度降低17.6%，服务等级目标（SLO）达成率提升最高20.1个百分点，且所有测试条件下令牌间隔延迟始终低于0.5毫秒，未改动传输协议、推理引擎或硬件。
原文摘要 · Abstract (English)
Disaggregated LLM inference forces the KV cache to traverse the datacenter network before decoding begins, so transfer time enters directly into the Time to First Token (TTFT) budget. Current schedulers route on compute load and prefix-cache locality alone, ignoring the topological distance and dynamic congestion between prefill and decode instances. We close this gap with a thin operator-to-scheduler interface, the network cost oracle, and we prove that ignoring the network term renders cache-aware-only scheduling arbitrarily suboptimal as context length grows. NetKV, the O(|D|) per-request greedy that consumes this oracle, has tier rankings that are provably robust to stale telemetry. On a 64-GPU four-tier fat-tree simulator driven by Mooncake traces, NetKV reduces mean TTFT by up to 21.2% over round-robin and 17.6% over a tuned cache+load-aware scheduler, lifts SLO attainment by up to 20.1 percentage points, and keeps the Time Between Tokens overhead below 0.5 ms in every condition tested, with no changes to the transport, inference engine, or hardware.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。