多级服务优先级调度让大模型推理更高效,兼顾不同用户需求。
Beyond Binary Priorities: Multi-Tier SLA Scheduling for Large Language Model Serving

- 支持任意数量优先级层级,动态分配计算资源
- 4个优先级时性价比最佳,吞吐提升最高达8.3倍
- 适合需要精细服务分级的生产环境部署
现代大模型服务需同时满足多样用户层级的差异化服务质量目标,从低延迟API请求到后台批量处理。现有方案Llumnix虽能实现负载均衡、碎片整理与自动扩缩容,但仅支持两级优先级,难以表达真实场景中复杂的SLA分类。本文将优先级模型扩展至任意层级,结合Vidur高保真模拟器,在三种典型优先级分布(均匀、正态、企业)下评估性能。引入分层预留空间(指数衰减)、层级感知任务调度及完整迁移流水线,集成于分层调度框架。对比INFaaS(全局路由)、vLLM、Orca和Sarathi-Serve(每副本基线),测试1~10级优先级。结果表明:4级优先级时成本-延迟比最优,预填充平均加速达8.3倍,端到端P99加速达3.1倍,单位延迟成本降低46%~68%,且各层级服务等级差异保持稳定。系统在10级优先级下仍无尾延迟崩溃,开销集中于预填充阶段。
原文摘要 · Abstract (English)
Modern LLM serving deployments must simultaneously satisfy heterogeneous service-level objectives (SLOs) across a diverse population of user tiers, ranging from latency-critical API calls to background batch processing. Llumnix introduced a dynamic, migration-capable multi-instance scheduler for LLM inference that achieves load balancing, defragmentation, prioritization, and auto-scaling through a unified "freeness" metric. However, Llumnix's priority model is restricted to two levels (high and normal), an abstraction too coarse to express the richer SLA classes common in production deployments. In this work, we extend Llumnix's priority model to support an arbitrary number of tiers and evaluate the effects of this extension under three realistic priority distributions (uniform, Gaussian, enterprise) using Vidur, a high-fidelity LLM inference simulator. We implement per-tier headroom with exponential decay, tier-aware dispatch ordering, and the full Llumnix migration pipeline inside Vidur's hierarchical scheduling framework. We compare our extended scheduler against INFaaS (global routing baseline), vLLM, Orca, and Sarathi-Serve (per-replica baselines), sweeping priority levels from 1 to 10. Our experiments demonstrate that four priority tiers yields the best cost-effectiveness tradeoff, achieving prefill mean speedups of up to 8.3x and end-to-end P99 speedups of up to 3.1x over INFaaS with cost-per-latency improvements of 46 to 68%, while preserving strong SLO differentiation across tiers. We further show that the system sustains these gains at 10 priority levels without tail latency collapse, with overhead concentrated in the prefill phase.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。