让在线离线任务共用资源，不降延迟还提吞吐
HyGen: Efficient LLM Serving via Elastic Online-Offline Request Co-location
- 通过预测延迟和量化干扰，动态控制资源分配
- 实测吞吐提升3.9至5.8倍，且在线延迟达标
- 适合需要高效利用算力的生产级LLM服务场景
大型语言模型（LLMs）支撑了从低延迟交互式聊天机器人到高吞吐量离线数据生成等多样应用，其服务等级目标（SLO）各异。现有部署方式将机器专用于不同工作负载，虽简化了SLO管理，但常导致资源利用率低下。本文提出HyGen，一种干扰感知的LLM服务系统，可在保障SLO的前提下，高效共置在线与离线任务。HyGen包含两项关键创新：(1) 性能控制机制，包括用于估计批处理执行时间的延迟预测器，以及用于量化延迟干扰的SLO感知剖析器；(2) SLO感知的离线调度策略，最大化服务吞吐并防止饥饿。在真实生产负载上的评估显示，相比在线与混合服务基线，HyGen可实现最高3.9至5.8倍的吞吐提升，同时确保延迟SLO达标。HyGen代码已公开于https://github.com/UIUC-MLSys/HyGen。
原文摘要 · Abstract (English)
Large language models (LLMs) have facilitated a wide range of applications with distinct service-level objectives (SLOs), from latency-sensitive online tasks like interactive chatbots to throughput-oriented offline workloads like data synthesis. The existing deployment model, which dedicates machines to each workload, simplifies SLO management but often leads to poor resource utilization. This paper introduces HyGen, an interference-aware LLM serving system that enables efficient co-location of online and offline workloads while preserving SLOs. HyGen incorporates two key innovations: (1) performance control mechanisms, including a latency predictor to estimate batch execution time and an SLO-aware profiler to quantify latency interference, and (2) SLO-aware offline scheduling policies that maximize serving throughput and prevent starvation. Our evaluation on production workloads shows that HyGen achieves up to 3.9-5.8x throughput gains over online and hybrid serving baselines, while ensuring latency SLOs. The code of HyGen is publicly available at https://github.com/UIUC-MLSys/HyGen.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。