arXiv:2606.22327cs.AI2026-06被引 3

针对大模型推理的内存调度难题,提出几何感知新算法,显著降低延迟。

Geometry-Aware Online Scheduling for LLM Serving: From Theoretical Bound to System Practice

论文配图:Geometry-Aware Online Scheduling for LLM Serving: From Theoretical Bound to System Practice
图 1 · 摘自论文原文
  • 基于体积最小优先原则,设计可理论证明的调度算法
  • 在高并发下将最坏情况竞争比从48降至3,性能大幅提升
  • 1比特信息即可实现高效调度,适合实际系统部署

大模型交互式推理对键值缓存动态内存占用的管理提出了严峻挑战。现有系统多采用时间为中心的调度策略(如最短作业优先),但其理论最优性基于传统调度模型,无法捕捉大模型推理中特有的二维时空动态增长特性。为此,我们提出几何感知在线调度,引入最小体积优先(SVF)算法及其高效变体1比特SVF。理论上,通过新颖的体积证书证明,将SVF在高并发场景下的最坏情况竞争比从先前最优的48提升至**3**。进一步构建了涵盖不同流量场景与信息可用性的完整理论分类体系。实践中,我们将该方法无缝集成至vLLM系统中。在Llama-3.1上的大量实验表明:SVF显著降低平均与尾部延迟;1比特SVF仅需单比特信息即达接近最优的吞吐与延迟表现。本工作为现代大模型部署中的内存受限调度提供了理论可靠且实证有效的解决方案。代码已开源:https://github.com/Aurora-Kl/Geometry-Aware-Online-Scheduling.git。

原文摘要 · Abstract (English)

The explosive demand for interactive Large Language Model serving has highlighted the management of the Key-Value cache's dynamic memory footprint as a critical area for performance optimization in inference engines. Modern inference systems overwhelmingly rely on time-centric scheduling heuristics, such as Shortest Job First. However, their theoretical optimality is rooted in traditional schedule modeling, failing to capture the highly dynamic, 2D spatio-temporal geometric growth specific to LLM inference mechanisms. To resolve this, we propose the geometry-aware online scheduling by introducing the Smallest Volume First (SVF) algorithm and its highly efficient variant, 1-bit SVF. Theoretically, we provide a rigorous mathematical foundation for our approach. Via a novel volume-certificate proof, we sharpen SVF's worst-case competitive ratio from the prior best of 48 towards \textbf{3} in the high-concurrency regime of LLM serving. Building upon this core breakthrough, we complete a comprehensive theoretical taxonomy analyzing our algorithms across different traffic scenarios and information availability. Practically, we seamlessly integrate our approach as a plug-and-play layer in vLLM. Extensive evaluations on Llama-3.1 models demonstrate comprehensive performance gains: SVF delivers strong reductions in both average and tail latency, while 1-bit SVF, with merely a single bit information, achieves competitive throughput and latency. This work establishes a theoretically sound and empirically proven approach for resolving memory-constrained scheduling in modern LLM deployments. To facilitate future research, our code is available at https://github.com/Aurora-Kl/Geometry-Aware-Online-Scheduling.git.

大模型推理调度算法内存优化在线调度

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。