针对多阶段大模型请求，实现按需分配算力以满足不同服务目标。
SLOs-Serve: Optimized Serving of Multi-SLO LLMs
- 基于动态规划算法，按阶段和应用定制令牌分配策略。
- 在6类场景下平均提升每张GPU的推理吞吐量2.2倍。
- 适合需要灵活支持多种服务质量要求的生产级大模型服务。
本文提出SLOs-Serve系统，用于服务具有应用级和阶段级服务等级目标（SLO）的多阶段大语言模型（LLM）请求。核心思想是根据SLO要求定制化分配令牌资源。SLOs-Serve采用基于多SLO动态规划的算法，在分块预填充和可选推测解码的完整设计空间中持续优化令牌分配，满足SLO约束。借助该资源规划算法，系统能有效支持多SLO与多副本服务，并实现动态请求路由，对突发流量具备鲁棒性。在6种典型大模型应用场景（包括摘要生成、代码生成、聊天机器人、工具调用和推理）上的评估表明，相比现有最先进系统，SLOs-Serve平均提升每张GPU的服务容量2.2倍。
原文摘要 · Abstract (English)
This paper introduces SLOs-Serve, a system designed for serving multi-stage large language model (LLM) requests with application- and stage-specific service level objectives (SLOs). The key idea behind SLOs-Serve is to customize the allocation of tokens to meet these SLO requirements. SLOs-Serve uses a multi-SLO dynamic programming-based algorithm to continuously optimize token allocations under SLO constraints by exploring the full design space of chunked prefill and (optional) speculative decoding. Leveraging this resource planning algorithm, SLOs-Serve effectively supports multi-SLOs and multi-replica serving with dynamic request routing while being resilient to bursty arrivals. Our evaluation across 6 LLM application scenarios (including summarization, coding, chatbot, tool calling, and reasoning) demonstrates that SLOs-Serve improves per-GPU serving capacity by 2.2x on average compared to prior state-of-the-art systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。