提出新调度算法，兼顾大模型推理的公平性与局部性
Locality-aware Fair Scheduling in LLM Serving
- 设计基于亏欠机制的前缀匹配调度，兼顾缓存局部性与客户端公平
- 单机版比VTC高2.87倍吞吐，客户端延迟低至最优系统的1/7.18
- 适用于多客户端共享算力场景，尤其适合长文本生成等任务
大型语言模型（LLM）推理负载广泛应用于多轮对话、文档分析等现代AI应用。在处理具有不同前缀模式的多样化客户端工作负载时，公平性与效率的平衡至关重要。现有公平调度算法如虚拟令牌计数器（VTC）忽略前缀局部性，性能不佳；而现有的局部性感知调度算法虽提升缓存命中率，却未考虑客户端间的公平共享。本文提出首个兼顾局部性的公平调度算法——亏欠最长前缀匹配（DLPM），可在保证公平性的同时维持高前缀局部性。进一步提出分布式扩展版本双亏欠LPM（D²LPM），在公平性、局部性和负载均衡间取得平衡。大量实验表明，DLPM和D²LPM在保障公平性的同时，吞吐量最高达VTC的2.87倍，单客户端延迟最低仅为当前最优分布式系统7.18倍。该研究显著提升了大规模语言模型服务的综合性能。
原文摘要 · Abstract (English)
Large language model (LLM) inference workload dominates a wide variety of modern AI applications, ranging from multi-turn conversation to document analysis. Balancing fairness and efficiency is critical for managing diverse client workloads with varying prefix patterns. Unfortunately, existing fair scheduling algorithms for LLM serving, such as Virtual Token Counter (VTC), fail to take prefix locality into consideration and thus suffer from poor performance. On the other hand, locality-aware scheduling algorithms in existing LLM serving frameworks tend to maximize the prefix cache hit rate without considering fair sharing among clients. This paper introduces the first locality-aware fair scheduling algorithm, Deficit Longest Prefix Match (DLPM), which can maintain a high degree of prefix locality with a fairness guarantee. We also introduce a novel algorithm, Double Deficit LPM (D$^2$LPM), extending DLPM for the distributed setup that can find a balance point among fairness, locality, and load-balancing. Our extensive evaluation demonstrates the superior performance of DLPM and D$^2$LPM in ensuring fairness while maintaining high throughput (up to 2.87$\times$ higher than VTC) and low per-client (up to 7.18$\times$ lower than state-of-the-art distributed LLM serving system) latency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。