提出两种易实现的调度方法，显著提升大模型服务吞吐量。
Is the GPU Half-Empty or Half-Full? Practical Scheduling Techniques for LLMs
- 设计轻量级调度策略，兼顾性能与部署复杂度
- 在真实负载下比现有方案提升20%以上吞吐量
- 适合追求稳定高效的大模型服务系统团队
大语言模型（LLM）服务系统通过并发处理多个请求来提升吞吐量，但如何在并发请求间合理调度硬件资源仍面临挑战。实际系统通常在两级实现调度决策：首先由负载均衡器将请求分配到不同服务器（每台服务器运行一个LLM副本），然后在每台服务器上由引擎级调度器决定请求的执行、排队或抢占时机。本文调研了学术界和工业界现有的调度技术，发现学术方案虽性能优异但复杂度高，而工业部署方案虽易用却常忽略明显性能提升空间。为此，我们提出两种新调度技术，在保持低实现成本的同时，优于现有生产环境工作负载上的主流策略。
原文摘要 · Abstract (English)
Serving systems for Large Language Models (LLMs) improve throughput by processing several requests concurrently. However, multiplexing hardware resources between concurrent requests involves non-trivial scheduling decisions. Practical serving systems typically implement these decisions at two levels: First, a load balancer routes requests to different servers which each hold a replica of the LLM. Then, on each server, an engine-level scheduler decides when to run a request, or when to queue or preempt it. Improved scheduling policies may benefit a wide range of LLM deployments and can often be implemented as "drop-in replacements" to a system's current policy. In this work, we survey scheduling techniques from the literature and from practical serving systems. We find that schedulers from the literature often achieve good performance but introduce significant complexity. In contrast, schedulers in practical deployments often leave easy performance gains on the table but are easy to implement, deploy and configure. This finding motivates us to introduce two new scheduling techniques, which are both easy to implement, and outperform current techniques on production workload traces.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。