首次系统分析vLLM冷启动延迟，揭示关键瓶颈并提供预测工具。
Breaking the Ice: Analyzing Cold Start Latency in vLLM
- 将冷启动拆解为六步，发现主要受CPU性能制约。
- 各步骤延迟随模型规模和硬件参数呈可预测规律变化。
- 构建轻量级模型，精准预测启动时间，助力大规模部署资源规划。
随着可扩展推理服务的普及，推理引擎的冷启动延迟日益重要。vLLM已成为众多推理任务的首选引擎。然而，由于其架构复杂且演进迅速，目前尚无对其启动延迟的系统性研究。本文针对vLLM的多项重大架构创新（如V1 API、torch.compile引入），首次开展详细的性能表征。我们将启动过程分解为六个基础步骤，证明该过程主要由CPU主导。每个步骤均表现出与模型级和系统级参数一致且可解释的缩放趋势，支持对延迟来源的细粒度归因。基于这些洞察，我们构建了一个轻量级解析模型，能准确预测给定硬件配置下的vLLM启动延迟，为大规模推理环境中的资源规划提供切实指导。所有基准数据集、分析工具及预测脚本均已开源至https://github.com/upb-cn/vllm-startup-profiler。
原文摘要 · Abstract (English)
As scalable inference services become popular, the cold start latency of an inference engine becomes important. Today, vLLM has evolved into the de-facto inference engine of choice for many inference workloads. Although popular, due to its complexity and rapid evolution, there has not been a systematic study on the startup latency of its engine. With major architectural innovations under it (e.g., the V1 API, introduction of torch.compile), in this paper, we present the first detailed performance characterization of vLLM startup latency. We break down the startup process into six foundational steps and demonstrate that this process is predominantly CPU-bound. Each step exhibits consistent and interpretable scaling trends with respect to model- and system-level parameters, enabling fine-grained attribution of latency sources. Building on these insights, we develop a lightweight analytical model that accurately predicts vLLM's startup latency for a given hardware configuration, providing actionable guidance for resource planning in large-scale inference environments. All our benchmarking datasets, analysis tools, and prediction scripts are open-sourced at https://github.com/upb-cn/vllm-startup-profiler
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。