提出新调度框架，让大模型服务更公平高效。
Equinox: Holistic Fair Scheduling in Serving Large Language Models
- 双计数器分离用户与资源视角，量化服务公平性
- 预测延迟和吞吐，实现1.3倍吞吐提升、60%低首字延迟
- 适合高并发大模型服务场景，兼顾公平与性能
现有大模型服务在公平性方面存在局限，本文提出双计数器框架：用户公平性计数器基于加权令牌和延迟衡量服务质量；资源公平性计数器通过吞吐量和GPU利用率评估运营效率。由于这些指标仅在执行后可得，导致调度困境。为此，我们引入确定性预测专家混合（MoPE）框架，预测用户感知延迟、输出令牌数、吞吐量及GPU利用率。基于预测结果计算统一的综合公平分，通过可调参数实现前瞻性公平调度。我们实现了Equinox开源系统，集成自适应批处理与无阻塞调度等优化。在真实生产负载（ShareGPT、LMSYS）与合成负载上评估表明，Equinox相较VTC系统实现最高1.3倍吞吐、60%更低的首字延迟、13%更高公平性，同时保持94% GPU利用率，证明在异构平台下公平性仍能有效维持。
原文摘要 · Abstract (English)
We address the limitations of current LLM serving with a dual-counter framework separating user and operator perspectives. The User Fairness Counter measures quality of service via weighted tokens and latency; the Resource Fairness Counter measures operational efficiency through throughput and GPU utilization. Since these metrics are only available post-execution, creating a scheduling paradox, we introduce a deterministic Mixture of Prediction Experts (MoPE) framework to predict user-perceived latency, output tokens, throughput, and GPU utilization. These predictions enable calculation of a unified Holistic Fairness score that balances both counters through tunable parameters for proactive fairness-aware scheduling. We implement this in Equinox, an open-source system with other optimizations like adaptive batching, and stall-free scheduling. Evaluations on production traces (ShareGPT, LMSYS) and synthetic workloads demonstrate Equinox achieves up to $1.3\times$ higher throughput, 60\% lower time-to-first-token latency, and 13\% higher fairness versus VTC while maintaining 94\% GPU utilization, proving fairness under bounded discrepancy across heterogeneous platforms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。