arXiv:2512.09472cs.DCcs.LG2025-12中稿 · ICML被引 3

通过预加载多模型参数,显著降低大模型服务的首次响应延迟。

WarmServe: Enabling One-for-Many GPU Prewarming for Multi-LLM Serving

  • 根据负载预测提前预热多个模型参数,避免请求突发时延迟。
  • 尾部首次令牌延迟降低50.8倍,请求吞吐量提升2.5倍。
  • 适合高并发、负载可预测的大模型服务场景。

在共享GPU集群中部署多个大语言模型是提升资源效率的关键策略。现有系统虽提高显卡利用率,却导致推理性能下降,尤其是首次令牌时间(TTFT)。我们发现这是由于缺乏对未来工作负载特征的认知。而真实场景中的大模型服务负载具有强周期性和长期可预测性。本文提出一拖多显卡预热机制,基于负载预测主动将多个模型参数加载至显卡。预热后的权重使系统在请求突增时能快速启动服务实例。我们设计并实现温服(WarmServe)系统,包含三项关键技术:(1)优化预热决策的模型放置算法,以最小化跨模型预热干扰;(2)将运行中显卡的空闲键值缓存空间重用于新模型预热;(3)高效的显卡内存切换机制用于张量管理。在真实数据集上的评估显示,相比最先进的自动伸缩系统,温服将尾部首次令牌延迟降低最多50.8倍,同时支持的请求吞吐量比显卡共享系统高出2.5倍。

原文摘要 · Abstract (English)

Deploying multiple models within shared GPU clusters is a key strategy to improve resource efficiency in large language model (LLM) serving. Existing multi-LLM serving systems improve GPU utilization at the cost of degraded inference performance, particularly time-to-first-token (TTFT). We attribute this degradation to the lack of awareness regarding future workload characteristics. In contrast, recent analyses have shown the strong periodicity and long-term predictability of real-world LLM serving workloads. In this paper, we propose one-for-many GPU prewarming, which proactively loads parameters from multiple models onto GPUs based on workload forecasts. These prewarmed weights enable the system to promptly instantiate serving instances upon encountering request bursts. We design and implement WarmServe, a multi-LLM serving system incorporating three key techniques: (1) a model placement algorithm that optimizes prewarming decisions to minimize cross-model prewarming interference, (2) a KV cache reservation strategy that repurposes idle KV cache space on running GPUs for prewarming new models, and (3) an efficient GPU memory switching mechanism for tensor management. Evaluation on real-world datasets shows that WarmServe reduces tail TTFT by up to 50.8$\times$ compared to the state-of-the-art autoscaling-based system, while supporting up to 2.5$\times$ higher request throughput than the GPU-sharing system.

大模型服务预热机制显卡优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。