ELIS通过预测响应长度优化调度，减少大模型推理等待时间。
ELIS: Efficient LLM Iterative Scheduling System with Response Length Predictor
- 用BGE模型训练响应长度预测器，提前估算推理时长。
- 采用改进的最短剩余时间优先调度，平均完成时间降低19.6%。
- 适用于高并发大模型服务场景，适合生产环境部署。
我们提出ELIS，一种用于大语言模型（LLM）的服务系统，其核心为迭代最短剩余时间优先（ISRTF）调度器，旨在高效管理剩余词元数最少的推理任务。现有LLM服务系统多采用先来先服务策略，易引发“队首阻塞”问题。为克服此限制，需预测LLM推理耗时并采用最短作业优先策略。然而由于LLM的自回归特性，推理延迟预测极具挑战。ELIS通过使用BGE模型——一种基于编码器的先进模型——训练响应长度预测器来应对该挑战。此外，我们设计了针对现有LLM迭代批处理优化的ISRTF调度策略。为评估工业级表现，我们基于真实用户LLM服务追踪记录模拟请求流，并在Kubernetes上实现ELIS为云原生调度系统以测试生产环境性能。实验结果表明，ISRTF将平均任务完成时间最多降低19.6%。
原文摘要 · Abstract (English)
We propose ELIS, a serving system for Large Language Models (LLMs) featuring an Iterative Shortest Remaining Time First (ISRTF) scheduler designed to efficiently manage inference tasks with the shortest remaining tokens. Current LLM serving systems often employ a first-come-first-served scheduling strategy, which can lead to the "head-of-line blocking" problem. To overcome this limitation, it is necessary to predict LLM inference times and apply a shortest job first scheduling strategy. However, due to the auto-regressive nature of LLMs, predicting the inference latency is challenging. ELIS addresses this challenge by training a response length predictor for LLMs using the BGE model, an encoder-based state-of-the-art model. Additionally, we have devised the ISRTF scheduling strategy, an optimization of shortest remaining time first tailored to existing LLM iteration batching. To evaluate our work in an industrial setting, we simulate streams of requests based on our study of real-world user LLM serving trace records. Furthermore, we implemented ELIS as a cloud-native scheduler system on Kubernetes to evaluate its performance in production environments. Our experimental results demonstrate that ISRTF reduces the average job completion time by up to 19.6%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。