让大模型代理程序运行更快，通过智能调度减少等待时间。
Autellix: An Efficient Serving Engine for LLM Agents as General Programs
- 将程序视为整体，根据已完成调用动态调度后续请求。
- 在相同延迟下，吞吐量提升4到15倍，优于vLLM等系统。
- 适合需要高效执行复杂任务的大模型应用开发者。
大型语言模型（LLM）应用正从简单聊天机器人演变为动态的通用代理程序，通过大量调用和输出文本实现推理、探索与复杂任务求解。然而，现有LLM服务系统忽略了程序间及调用间的依赖关系，错失了优化机会。分析显示，提交至LLM服务引擎的程序面临长时间累积等待，主要源于单个请求和程序层面的队头阻塞。为此，我们提出Autellix，一种将程序视为第一类公民的LLM服务系统，以最小化端到端延迟。Autellix拦截程序提交的LLM调用，并向调度器注入程序级上下文。我们设计了两种调度算法——适用于单线程与分布式程序——基于程序先前完成的调用进行抢占和优先级排序。评估表明，在多种LLM和代理工作负载下，Autellix相比当前最先进的系统（如vLLM），在相同延迟条件下将程序吞吐量提升4至15倍。
原文摘要 · Abstract (English)
Large language model (LLM) applications are evolving beyond simple chatbots into dynamic, general-purpose agentic programs, which scale LLM calls and output tokens to help AI agents reason, explore, and solve complex tasks. However, existing LLM serving systems ignore dependencies between programs and calls, missing significant opportunities for optimization. Our analysis reveals that programs submitted to LLM serving engines experience long cumulative wait times, primarily due to head-of-line blocking at both the individual LLM request and the program. To address this, we introduce Autellix, an LLM serving system that treats programs as first-class citizens to minimize their end-to-end latencies. Autellix intercepts LLM calls submitted by programs, enriching schedulers with program-level context. We propose two scheduling algorithms-for single-threaded and distributed programs-that preempt and prioritize LLM calls based on their programs' previously completed calls. Our evaluation demonstrates that across diverse LLMs and agentic workloads, Autellix improves throughput of programs by 4-15x at the same latency compared to state-of-the-art systems, such as vLLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。