通过解耦抢占粒度与调度频率，缓解大模型服务中的首包延迟阻塞问题。
FlowPrefill: Decoupling Preemption from Prefill Scheduling Granularity to Mitigate Head-of-Line Blocking in LLM Serving
- 在算子边界实现细粒度抢占，避免固定小分块导致的效率损失。
- 仅在请求到达或完成时触发调度，降低控制开销并提升响应速度。
- 实测可提升最大吞吐5.6倍，适合高并发多优先级场景使用。
大语言模型（LLM）服务需处理大量并发请求，且存在多样化的服务等级目标（SLO）。这加剧了计算密集型预填充阶段的头部阻塞问题：长任务占用资源，延迟高优先级请求，导致广泛的时间到首个标记（TTFT）SLO违反。虽然分块预填充可实现可中断性，但存在响应性与吞吐量的固有权衡——减小分块提升响应延迟但降低计算效率，增大分块则最大化吞吐但加重阻塞。因此需要自适应抢占机制。然而，动态平衡执行粒度与调度开销仍是关键挑战。本文提出FlowPrefill，一种优化TTFT与吞吐量的推理系统，通过解耦抢占粒度与调度频率解决该矛盾。其核心创新包括：1）算子级抢占，利用算子边界实现细粒度执行中断，避免固定小分块带来的效率损失；2）事件驱动调度，仅在请求到达或完成时触发调度决策，实现高效响应的同时最小化控制平面开销。基于真实生产负载的评估显示，相比现有最优系统，FlowPrefill可将最大吞吐提升达5.6倍，同时满足异构SLO要求。
原文摘要 · Abstract (English)
The growing demand for large language models (LLMs) requires serving systems to handle many concurrent requests with diverse service level objectives (SLOs). This exacerbates head-of-line (HoL) blocking during the compute-intensive prefill phase, where long-running requests monopolize resources and delay higher-priority ones, leading to widespread time-to-first-token (TTFT) SLO violations. While chunked prefill enables interruptibility, it introduces an inherent trade-off between responsiveness and throughput: reducing chunk size improves response latency but degrades computational efficiency, whereas increasing chunk size maximizes throughput but exacerbates blocking. This necessitates an adaptive preemption mechanism. However, dynamically balancing execution granularity against scheduling overheads remains a key challenge. In this paper, we propose FlowPrefill, a TTFT-goodput-optimized serving system that resolves this conflict by decoupling preemption granularity from scheduling frequency. To achieve adaptive prefill scheduling, FlowPrefill introduces two key innovations: 1) Operator-Level Preemption, which leverages operator boundaries to enable fine-grained execution interruption without the efficiency loss associated with fixed small chunking; and 2) Event-Driven Scheduling, which triggers scheduling decisions only upon request arrival or completion events, thereby supporting efficient preemption responsiveness while minimizing control-plane overhead. Evaluation on real-world production traces shows that FlowPrefill improves maximum goodput by up to 5.6$\times$ compared to state-of-the-art systems while satisfying heterogeneous SLOs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。