让解码节点帮忙处理预填充,显著降低大模型服务延迟。
Towards Load-Aware Prefill Deflection for Disaggregated LLM Serving

- 解码节点分担预填充任务,避免预填充节点过载。
- 在2P2D集群上,尾部延迟(P95 TTFT)降低81%。
- 适合高并发、突发负载下的大规模大模型部署。
解耦式大模型服务将预填充和解码分别运行在独立的GPU池中,以避免相互干扰。但在实际应用中,这种设计引入了新问题:在突发性、长尾工作负载下,预填充节点会饱和,而解码节点计算资源大量闲置。在包含2个预填充节点和2个解码节点(2P2D)的A100生产集群上,预填充执行仅占总时间的2-23%。排队等待和跨节点的KV缓存传输占用了剩余时间。本文提出一种主动的预填充分流调度器,使解码节点可交错执行请求的分块预填充步骤,与自身正在进行的解码批处理并行。对每个排队请求,估算其在预填充节点上的时延,并在每个解码节点上寻找最大可调度块,确保在飞行中的解码任务保持在时间间隔(TBT)SLO范围内。当解码路径能改善尾部延迟时,便触发分流。由于被分流的请求预填充阶段直接在解码节点上执行,消除了跨节点的KV缓存传输。该方案在vLLM上实现,并基于DeepSeek-V2-Lite的真实生产轨迹进行评估,相比现有最优解耦调度器,将P95 TTFT降低高达81%,提升SLO达标率最高79%,且每请求路由开销低于1毫秒。
原文摘要 · Abstract (English)
Disaggregated LLM serving runs prefill and decode on separate GPU pools to keep the two phases from interfering. In practice, this creates a new asymmetry: under bursty, heavy-tailed workloads prefill nodes saturate while decode nodes have compute underutilized, and on a production-style A100 cluster with 2 prefill and 2 decode nodes (2P2D), we find that prefill execution accounts for only 2-23% of P95 Time-to-First-Token (TTFT). Queuing and inter-node GPU-GPU KV-cache transfer account for the rest. We present a proactive prefill-deflecting scheduler that lets decode nodes serve prefill phase of requests as chunked-prefill steps interleaved with their in-flight decode batches. For each queued request, we estimate the TTFT it would see on the prefill node, and on every decode node, search for the largest chunk schedule that keeps in-flight decodes within their Time-Between-Tokens (TBT) SLO and deflect when the decode path helps tail latency. Because the prefill phase of deflected requests runs in place on the decode node, the inter-node KV transfer is eliminated. Implemented on vLLM and evaluated on production-style traces with DeepSeek-V2-Lite, our approach reduces P95 TTFT by upto 81% and raises SLO attainment by upto 79% over state-of-the-art disaggregated schedulers, at sub-millisecond per-request routing cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。