AdaServe通过定制化推测解码，实现多延迟目标下高效大模型服务。
AdaServe: Accelerating Multi-SLO LLM Serving with SLO-Customized Speculative Decoding
- 为不同延迟需求定制推测解码树，动态调整参数应对负载变化。
- 相比最优基线，延迟违规减少4.3倍，吞吐量提升1.9倍。
- 适合对响应速度有差异要求的复杂大模型应用，如交互式编程与数据处理。
现代大语言模型应用具有多样化的服务级别目标（SLO），从交互式编程助手的低延迟需求到数据清洗任务的宽松约束。现有大模型服务系统依赖统一批处理和调度策略，难以同时满足这些异构SLO。我们提出AdaServe，首个支持高效多SLO服务的大模型服务系统，采用定制化推测解码机制。AdaServe将多SLO服务建模为带约束的优化问题，设计硬件感知算法，为每项请求构建匹配其延迟目标的推测树。系统采用推测-选择-验证流水线，实现解码速度的细粒度控制，同时最大化吞吐量。此外，通过动态调整推测参数适应负载变化。在多种工作负载下的评估显示，与最优基线相比，AdaServe将SLO违规降低4.3倍，好吞吐量提升1.9倍，显著提升了多SLO服务的效率。
原文摘要 · Abstract (English)
Modern large language model (LLM) applications exhibit diverse service-level objectives (SLOs), from low-latency requirements in interactive coding assistants to more relaxed constraints in data wrangling tasks. Existing LLM serving systems, which rely on uniform batching and scheduling strategies, often fail to meet these heterogeneous SLOs concurrently. We present AdaServe, the first LLM serving system designed to support efficient multi-SLO serving through SLO-customized speculative decoding. AdaServe formulates multi-SLO serving as a constrained optimization problem and introduces a hardware-aware algorithm that constructs a speculation tree tailored to each request's latency target. It features a speculate-select-verify pipeline that enables fine-grained control over decoding speed while maximizing system throughput. AdaServe further adapts to workload variation by dynamically adjusting speculation parameters. Evaluations across diverse workloads show that AdaServe reduces SLO violations by up to 4.3$\times$ and improves goodput by up to 1.9$\times$ compared to the best performing baselines, highlighting its effectiveness in multi-SLO serving.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。