为混合优先级的MoE模型推理设计了细粒度抢占调度，显著降低响应延迟。
Priority-Aware Preemptive Scheduling for Mixed-Priority Workloads in MoE Inference
- 基于专家层级的优先级抢占调度，突破传统迭代级排队限制。
- 低延迟任务首令牌时间减少65.5倍，支持每秒7个请求达标服务等级。
- 适用于需要高响应速度的生产级大模型服务场景。
大规模语言模型革新了自然语言处理，但在数据中心高效服务仍面临挑战，因混合负载包含对延迟敏感（LS）和尽力而为（BE）的任务。现有推理系统采用迭代级先来先服务调度，导致BE任务阻塞LS任务的首令牌时间。本文提出QLLM，一种专为混合专家（MoE）模型设计的新型推理系统，具备细粒度、优先级感知的抢占式调度器。该系统支持专家层级的抢占，可推迟BE任务执行，同时最小化LS任务的首令牌时间。其方法打破迭代级调度约束，允许根据优先级在任意层进行任务抢占。在Nvidia A100 GPU上的评估显示，QLLM显著提升性能：平均将LS首令牌时间缩短65.5倍，最高可在7请求/秒下满足服务等级目标，而基线系统无法达成；同时，LS任务周转时间最多缩短12.8倍，且不影响吞吐量。QLLM模块化、可扩展，能无缝集成至Hugging Face MoE模型。
原文摘要 · Abstract (English)
Large Language Models have revolutionized natural language processing, yet serving them efficiently in data centers remains challenging due to mixed workloads comprising latency-sensitive (LS) and best-effort (BE) jobs. Existing inference systems employ iteration-level first-come-first-served scheduling, causing head-of-line blocking when BE jobs delay LS jobs. We introduce QLLM, a novel inference system designed for Mixture of Experts (MoE) models, featuring a fine-grained, priority-aware preemptive scheduler. QLLM enables expert-level preemption, deferring BE job execution while minimizing LS time-to-first-token (TTFT). Our approach removes iteration-level scheduling constraints, enabling the scheduler to preempt jobs at any layer based on priority. Evaluations on an Nvidia A100 GPU show that QLLM significantly improves performance. It reduces LS TTFT by an average of $65.5\times$ and meets the SLO at up to $7$ requests/sec, whereas the baseline fails to do so under the tested workload. Additionally, it cuts LS turnaround time by up to $12.8\times$ without impacting throughput. QLLM is modular, extensible, and seamlessly integrates with Hugging Face MoE models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。