用预测调度提升大模型服务吞吐，缓解排队延迟问题
ALISE: Accelerating Large Language Model Serving with Speculative Scheduling
- 根据任务执行时间预估，动态调整优先级避免队列阻塞
- 在Alpaca和ShareGPT数据集上，吞吐提升1.8倍至2.1倍
- 适合高并发、异构负载的大模型推理服务场景
大语言模型（LLM）是当前通用人工智能的重要进展。以ChatGPT为代表的基于LLM的应用要求极低响应延迟和最高吞吐量。然而，由于LLM执行过程的不可预测性，现有系统采用的先到先服务（FCFS）调度策略存在首部阻塞（HoL）问题，导致长作业响应时间。本文提出新型高效的LLM推理服务框架ALISE，其核心设计是通过新颖的推测式调度器，预估每个任务的执行时间，并利用该先验知识分配合适的优先级顺序，从而最小化异构工作负载下的潜在排队延迟。此外，为降低中间键值（KV）缓存的内存开销，采用基于优先级的自适应内存管理协议与量化压缩技术。评估表明，在相同延迟约束下，相比最先进的vLLM方案，ALISE在Alpaca和ShareGPT数据集上的推理吞吐分别提升了1.8倍和2.1倍。
原文摘要 · Abstract (English)
Large Language Models (LLMs) represent a revolutionary advancement in the contemporary landscape of artificial general intelligence (AGI). As exemplified by ChatGPT, LLM-based applications necessitate minimal response latency and maximal throughput for inference serving. However, due to the unpredictability of LLM execution, the first-come-first-serve (FCFS) scheduling policy employed by current LLM serving systems suffers from head-of-line (HoL) blocking issues and long job response times. In this paper, we propose a new efficient LLM inference serving framework, named ALISE. The key design paradigm of ALISE is to leverage a novel speculative scheduler by estimating the execution time for each job and exploiting such prior knowledge to assign appropriate job priority orders, thus minimizing potential queuing delays for heterogeneous workloads. Furthermore, to mitigate the memory overhead of the intermediate key-value (KV) cache, we employ a priority-based adaptive memory management protocol and quantization-based compression techniques. Evaluations demonstrate that in comparison to the state-of-the-art solution vLLM, ALISE improves the throughput of inference serving by up to 1.8x and 2.1x under the same latency constraint on the Alpaca and ShareGPT datasets, respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。