动态调整请求优先级，让大模型服务同时快响应又高吞吐。
Ascendra: Dynamic Request Prioritization for Efficient LLM Serving
- 按请求紧迫度动态分配高低优先级实例
- 吞吐量最高提升1.7倍，且满足双延迟指标
- 适合对响应速度和处理效率都要求高的场景
大语言模型（LLM）的快速发展推动了更高效的部署需求。当前效率主要体现在满足服务等级目标（SLO）的比例上，尤其是首令牌时间（TTFT）和令牌间时间（TBT）。现有系统常在两者之间取舍。本文提出Ascendra，一种能同时满足TTFT和TBT SLO的LLM服务系统。核心思想是：请求的紧迫性随截止时间临近而变化。为此，Ascendra将GPU资源划分为低优先级与高优先级实例：低优先级实例按到达顺序处理以最大化吞吐，但存在饥饿风险；通过性能模型预测可能超时的请求并主动迁移至高优先级实例，后者专为低延迟优化，处理临近截止的紧急请求。该分层架构有效平衡了高吞吐与低延迟。大量实验表明，相较于vLLM和Sarathi-Serve，Ascendra在满足双SLO的前提下，系统吞吐最高提升1.7倍。
原文摘要 · Abstract (English)
The rapid advancement of Large Language Models (LLMs) has driven the need for more efficient serving strategies. In this context, efficiency refers to the proportion of requests that meet their Service Level Objectives (SLOs), particularly for Time To First Token (TTFT) and Time Between Tokens (TBT). However, existing systems often prioritize one metric at the cost of the other. We present Ascendra, an LLM serving system designed to meet both TTFT and TBT SLOs simultaneously. The core insight behind Ascendra is that a request's urgency evolves as it approaches its deadline. To leverage this, Ascendra partitions GPU resources into two types of instances: low-priority and high-priority. Low-priority instances maximize throughput by processing requests out of arrival order, but at the risk of request starvation. To address this, Ascendra employs a performance model to predict requests at risk of missing their SLOs and proactively offloads them to high-priority instances. High-priority instances are optimized for low-latency execution and handle urgent requests nearing their deadlines. This partitioned architecture enables Ascendra to effectively balance high throughput and low latency. Extensive evaluation shows that Ascendra improves system throughput by up to 1.7x compared to vLLM and Sarathi-Serve while meeting both TTFT and TBT SLOs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。