通过内存弹性分配，让多个大模型共用显存，降低成本。
Prism: Cost-Efficient Multi-LLM Serving via GPU Memory Ballooning
- 用动态内存回收技术统一实现显存和时间共享
- 在1万+块GPU上部署，显著提升资源利用率
- 适合需要低成本运行多个模型的团队使用
推理服务提供商需保障多种大模型的可用性，包括低频但关键的模型。随着每令牌价格下降，资源效率愈发重要。对生产环境日志的分析发现，模型组呈现动态突发性活跃模式，即多模型同时启停并随时间变化；现有空间与时间共享方法缺乏应对这种变化的机制，导致服务质量目标（SLO）与效率之间难以兼顾。我们观察到，弹性内存分配可统一实现空间与时间共享。基于此，提出Prism——一种以内存为中心的大模型共服务框架，通过内存膨胀技术跨模型回收内存，用单一方案支持两类共享。其核心内存驱动kvcached已开源（https://github.com/ovg-project/kvcached），并在超1万块GPU的生产环境中部署。
原文摘要 · Abstract (English)
Inference providers must maintain availability for many LLMs, including low-volume but essential models, making resource efficiency increasingly important as token prices fall. Analysis of production traces reveals a dynamic bursty-group pattern in which sets of models become active together and shift over time; existing space- and time-sharing approaches lack principled mechanisms to adapt to this variability, forcing trade-offs between SLO adherence and efficiency. We observe that elastic memory allocation can unify spatial and temporal sharing. Based on this insight, we have developed Prism, a memory-centric LLM co-serving framework that applies memory ballooning to reclaim memory across models and support both forms of sharing under a single scheme. Prism's balloon driver, referred to as kvcached, has been open-sourced at https://github.com/ovg-project/kvcached, and deployed in production environments across 10K+ GPUs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。