通过重用GPU内存，让服务器无服务器大模型加载快6.2倍。
Tangram: Accelerating Serverless LLM Loading through GPU Memory Reuse and Affinity
- 用统一内存池共享模型参数，减少重复加载
- 冷启动时加载速度提升6.2倍，首令牌延迟降23%~55%
- 适合追求快速响应的云端大模型服务部署
无服务器大语言模型（LLM）通过共享GPU资源实现按需付费，但冷启动延迟（尤其是模型加载阶段）随模型规模线性增长，成为实际部署的主要瓶颈。本文提出Tangram系统，通过高效利用未使用的GPU内存保留模型参数，显著降低模型传输时间和冷启动延迟。其设计包含三个核心组件：跨模型的张量级参数共享统一内存池、按需分配的KV缓存动态管理机制，以及面向GPU亲和性的调度策略。这些技术共同解决了无服务器平台中内存利用率低与冷启动问题。我们实现了完整原型，实验表明，相较现有最优方法，Tangram在加载速度上最高提升6.2倍，冷启动时首令牌延迟（TTFT）降低23%至55%。
原文摘要 · Abstract (English)
Serverless Large Language Models (LLMs) have emerged as a cost-effective solution for deploying AI services by enabling a 'pay-as-you-go' pricing model through GPU resource sharing. However, cold-start latency, especially the model loading phase, has become a critical performance bottleneck, as it scales linearly with model size and severely limits the practical deployment of large-scale LLM services. This paper presents Tangram, a novel system that accelerates Serverless LLM loading through efficient GPU memory reuse. By leveraging the unused GPU memory to retain model parameters, Tangram significantly reduces model transfer time and cold-start latency. Its design includes three key components: unified GPU memory pool for tensor-level parameter sharing across models, on-demand KV cache allocation for dynamic memory management, and GPU-affinity-aware scheduling for maximizing resource utilization. These techniques collectively address the critical challenges of inefficient memory usage and the cold-start problem in Serverless LLM platforms. We have implemented a fully functional prototype, and experiments show that Tangram achieves up to 6.2 times faster loading and reduces Time-To-First-Token (TTFT) during cold-start by 23--55% over state-of-the-art methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。