针对大模型服务中不同请求的时效要求，动态调度提升吞吐与达标率。
Scorpio: Serving Right Requests at the Right Time for Heterogeneous SLOs in LLM Inference
- 按请求时效需求动态调度，分阶段优化准入与批处理。
- 高负载下吞吐量最高提升14.4倍，服务达标率提高46.5%。
- 适合对响应速度有差异化要求的在线大模型应用。
大型语言模型（LLM）服务正支撑着对话代理、网络搜索和编程助手等在线服务，其中请求具有异构的服务等级目标（SLO），如首次输出时间（TTFT）和每输出词元耗时（TPOT）。现有LLM服务系统以最大化吞吐量为目标，对所有请求一视同仁，导致SLO达成不佳。本文提出Scorpio，一种面向SLO的大模型服务系统，旨在最大化系统良好吞吐量与各类SLO的达成率。核心思想是利用SLO差异性，在准入控制、队列管理与批处理选择中实现自适应调度。Scorpio包含TTFT守卫，采用最迟截止时间优先重排序并拒绝无法满足的请求；以及TPOT守卫，结合基于VBS的准入控制与创新的信用批处理机制。两者均由预测模块支持。评估表明，相较于最先进基线，Scorpio在高负载下系统良好吞吐量最高提升14.4倍，SLO达标率最高提升46.5%。
原文摘要 · Abstract (English)
Large Language Model (LLM) serving increasingly underpins online Web services such as conversational agents, Web search, and programming assistants, where requests carry heterogeneous Service Level Objectives (SLOs) such as Time to First Token (TTFT) and Time Per Output Token (TPOT). Existing LLM serving systems prioritize maximum throughput and treat all requests uniformly, which leads to suboptimal SLO attainment. This paper introduces Scorpio, an SLO-oriented LLM serving system designed to maximize system goodput and SLO attainment for workloads with heterogeneous SLOs. Our core insight is to exploit SLO heterogeneity for adaptive scheduling across admission control, queue management, and batch selection. Scorpio features a TTFT Guard, which employs least-deadline-first reordering and rejects unattainable requests, and a TPOT Guard, which utilizes a VBS-based admission control and a novel credit-based batching mechanism. Both guards are supported by a predictive module. Evaluations demonstrate that Scorpio improves system goodput by up to 14.4x and SLO adherence by up to 46.5% under high load compared to state-of-the-art baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。