Chiron通过分层反馈机制，智能调节大模型服务资源，提升响应速度和硬件效率。
Hierarchical Autoscaling for Large Language Model Serving with Chiron
- 根据请求延迟要求分层管理资源，利用队列长度和利用率动态调整
- 在实验中实现90%更高的服务达标率，GPU使用效率提升70%
- 适合需要高响应性与资源效率的云服务场景
大型语言模型（LLM）推理服务正成为云服务商的重要负载。根据性能服务级别目标（SLO），LLM请求可分为两类：交互式请求（延迟要求在秒级）和批量请求（延迟要求在分钟至小时级）。这些SLO会因请求到达率、请求合并程度及配置参数变化而下降，因此需对服务实例及其批处理大小进行资源自动扩展。然而，现有LLM服务自动扩展器未考虑请求的SLO差异，导致过度扩展和资源闲置。为此，我们提出Chiron，一种基于队列大小、资源利用率和SLO的分层反压机制的自动扩展器。实验表明，相比现有方案，Chiron在服务达标率上最高提升90%，并使GPU效率提高最高达70%。
原文摘要 · Abstract (English)
Large language model (LLM) serving is becoming an increasingly important workload for cloud providers. Based on performance SLO requirements, LLM inference requests can be divided into (a) interactive requests that have tight SLOs in the order of seconds, and (b) batch requests that have relaxed SLO in the order of minutes to hours. These SLOs can degrade based on the arrival rates, multiplexing, and configuration parameters, thus necessitating the use of resource autoscaling on serving instances and their batch sizes. However, previous autoscalers for LLM serving do not consider request SLOs leading to unnecessary scaling and resource under-utilization. To address these limitations, we introduce Chiron, an autoscaler that uses the idea of hierarchical backpressure estimated using queue size, utilization, and SLOs. Our experiments show that Chiron achieves up to 90% higher SLO attainment and improves GPU efficiency by up to 70% compared to existing solutions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。