针对大模型代理并行任务，提出高效公平的调度器，提升完成速度且保障最差情况延迟。
Justitia: Fair and Efficient Scheduling of Task-parallel LLM Agents with Selective Pampering
- 按内存占用量化代理真实成本，实现精准调度
- 预测成本准确且轻量，支持快速决策
- 基于虚拟时间的公平队列，兼顾效率与公平
大模型代理常由并行推理任务构成，广泛用于解决实际问题。在共享GPU服务器上服务此类任务并行的大模型代理时，调度器需在保证最差情况延迟的前提下实现快速代理完成。我们的核心洞察是：根据理想化公平共享下的完成顺序，对代理进行选择性优待。为此，我们设计了Justitia，一种面向任务并行大模型代理的高效且公平的调度器。考虑到内存在大模型服务中普遍为瓶颈，Justitia以内存为中心量化代理的真实成本，并采用轻量但准确的方法预测代理成本。最后，通过基于虚拟时间的公平队列算法，降低整体延迟并保证最差情况延迟。我们在vLLM之上实现了Justitia，实验结果表明，其在多种代理场景下显著提升了调度效率，同时保持了公平性。
原文摘要 · Abstract (English)
LLM agents, which often comprise parallel inference tasks, are commonly adopted to solve real-world problems. When serving such task-parallel LLM agents in shared GPU servers, the scheduler is expected to attain fast agent completion with guaranteed worst-case performance. For that objective, our insight is to selectively pampering agents based on their completion order under idealized fair-sharing. We design Justitia, a fair and also efficient scheduler for task-parallel LLM agents. Noticing that memory is prevalently a bottleneck in LLM serving, Justitia quantifies the true agent cost in a memory-centric manner. It also adopts a light-weight yet accurate method to predict agent costs. Finally, Justitia adopts a virtual-time based fair queuing algorithm to reduce the overall performance with guaranteed worst-case delay. We have implemented Justitia atop vLLM, and experimental results involving diverse agents show that it can substantially enhance the scheduling efficiency with fairness preserved.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。