针对大模型服务中不同长度请求导致的延迟问题,提出按长度拆分调度的优化系统。
LAPS: A Length-Aware-Prefill LLM Serving System
- 按提示词长度拆分长/短请求,动态调度避免性能瓶颈。
- 在多轮场景下降低预填充延迟超30%,减少28%服务等级违规。
- 适合高并发、混合请求的大模型部署,尤其优化长文本生成场景。
LAPS 通过识别并拆分不同提示词长度的请求,降低大模型服务中的首字延迟(TTFT)。尽管现有系统已将预填充与解码阶段解耦以提升吞吐量,但仍采用统一调度策略,无法适应异构工作负载特性。研究发现,提示词长度差异会引发不同的性能瓶颈,因此提出自适应调度机制。LAPS 将多轮长提示请求与短提示请求分离,并为短提示任务引入基于长度感知的智能批处理。采用双队列设计,支持单实例的时间拆分或跨实例的空间拆分。对于短提示批处理,通过批等待窗口与基于 CUDA Graph 的聚类技术,缓解异构计算带来的干扰,降低批处理延迟,减少平均延迟。在真实多轮负载下,相比基础 SGLang 在预填充-解码解耦配置中,预填充延迟降低超过30%;在多实例部署中,服务等级协议(SLO)违规减少28%。相较于带负载均衡的 SGLang 路由器,在多 GPU 环境下进一步降低 SLO 违规12%。在高并发与混合请求场景下,服务 Qwen2.5-32B 模型时,预填充实例的请求吞吐量提升35%,验证了其在优化异构大模型服务负载方面的有效性。
原文摘要 · Abstract (English)
LAPS identifies and disaggregates requests with different prompt lengths in LLM serving to reduce TTFT latency. While recent systems have decoupled the prefill and decode stages to improve throughput, they still rely on unified scheduling policies that fail to adapt to heterogeneous workload characteristics. We observe that prompt-length variations lead to distinct performance bottlenecks, motivating an adaptive scheduling strategy. LAPS disaggregates multi-turn long-prefill requests from short-prefill ones and introduces a length-aware smart batching mechanism for short-prefill workloads. It adopts a dual-queue design that supports temporal disaggregation on a single prefill instance or spatial disaggregation across multiple instances. For short-prefill batches, a batch waiting window and CUDA Graph-based clustering mitigate interference from heterogeneous computation, reducing batching delay and lowering average latency. In real multi-turn workloads, LAPS reduces prefill latency by over 30\% compared to vanilla SGLang under prefill-decode disaggregation, and further decreases SLO violations by 28\% in multi-instance deployments with vanilla data-parallel configuration. Compared to the SGLang router with load balancing, it further lowers SLO violations by 12\% in multi-GPU settings. Under high concurrency and mixed-request scenarios, LAPS improves request throughput by 35\% serving Qwen2.5-32B model for prefill instance, demonstrating its effectiveness in optimizing heterogeneous LLM serving workloads.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。