动态调整推理任务调度，降低延迟并提升多租户服务稳定性。
DriftSched: Adaptive QoS-Aware Scheduling under Runtime Token Drift for Multi-Tenant GPU Inference
- 根据运行时反馈实时校准任务估算，减少误分类
- 自适应校准使估计误差降低38.8%（MAE）和40.5%（RMSE）
- 最短作业优先（SJF）策略显著降低尾部延迟，适合高并发场景
大语言模型（LLM）推理服务的快速发展推动了高效多租户GPU调度的需求。尽管vLLM等现代推理运行时通过连续批处理和优化内存管理提升了吞吐量，但对异构推理请求的运行时成本准确估计仍具挑战。实际中，准入时的工作负载估算常与执行行为偏差，导致任务误分类、队列失衡、尾部延迟上升和服务质量下降。本文提出DriftSched，一种针对NVIDIA L4 GPU的多租户LLM推理服务的QoS感知调度框架。该框架结合工作负载分类、令牌预算估算、租户感知队列管理及在线反馈机制，利用运行时观测结果修正工作负载估计。在多种调度策略（FIFO、优先级、加权、最短作业优先、老化优先）下评估异构多租户负载表现。实验表明，自适应校准使平均估计误差降低38.8%（MAE）和40.5%（RMSE），提升分类稳定性。所有策略中，最短作业优先（SJF）表现最优，相比FIFO在持续GPU争用下，中位端到端延迟降低约42%，P99延迟降低约16%。结果还表明，调度器选择对延迟行为的影响大于仅依赖运行时校准，而精准的工作负载表征可基本消除系统性估算漂移。本工作提供了一个可复现的框架，用于研究多租户GPU推理中的工作负载估计保真度与QoS感知调度。
原文摘要 · Abstract (English)
The rapid growth of large language model (LLM) inference services has increased the demand for efficient multi-tenant GPU scheduling. While modern inference runtimes such as vLLM improve throughput through continuous batching and optimized memory management, accurately estimating the runtime cost of heterogeneous inference requests remains challenging. In practice, admission-time workload estimates may deviate from observed execution behavior, leading to workload misclassification, queue imbalance, increased tail latency, and degraded Quality-of-Service (QoS). This paper presents DriftSched, a QoS-aware scheduling framework for multi-tenant LLM inference serving on NVIDIA L4 GPUs. DriftSched combines workload classification, token-budget estimation, tenant-aware queue management, and an online feedback mechanism to refine workload estimates using runtime observations. The framework evaluates FIFO, Priority, Weighted, Shortest-Job-First (SJF), and Aging Priority scheduling policies under heterogeneous multi-tenant workloads. Experimental results show that adaptive calibration reduces workload estimation error by an average of 38.8% (MAE) and 40.5% (RMSE), improving workload classification stability. Among all evaluated schedulers, SJF achieves the best overall performance, reducing median end-to-end latency by approximately 42% and P99 latency by approximately 16% relative to FIFO under sustained GPU contention. The results further indicate that scheduler selection has a greater impact on latency behavior than runtime calibration alone, while accurate workload characterization largely eliminates systematic estimation drift. This work contributes a reproducible framework for studying workload-estimation fidelity and QoS-aware scheduling in multi-tenant GPU inference systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。