通过缓存卸载优化云上大模型推理，显著降低部署成本。
Cost-Efficient LLM Serving in the Cloud: VM Selection with KV Cache Offloading
- 根据服务目标和负载特征动态调整键值缓存卸载策略。
- 在线场景下降本最高达73.7%，离线场景节省最多20.19%。
- 适合关注云上大模型部署成本的工程师与架构师。
大语言模型推理在文本摘要、翻译和数据分析等应用中至关重要，但云服务商（如AWS）的GPU实例高昂成本构成主要负担。本文提出InferSave框架，通过基于服务等级目标（SLOs）和工作负载特性优化键值（KV）缓存卸载，估算GPU内存需求，并推荐更具成本效益的虚拟机实例。此外，计算时间校准函数（CTCF）通过修正理论性能与实际表现之间的差异，提升实例选择精度。在AWS GPU实例上的实验表明，不采用KV缓存卸载而选择低成本实例，可使在线工作负载成本效率提升最高达73.7%；而在离线工作负载中，启用缓存卸载可节省最多20.19%的成本。
原文摘要 · Abstract (English)
LLM inference is essential for applications like text summarization, translation, and data analysis, but the high cost of GPU instances from Cloud Service Providers (CSPs) like AWS is a major burden. This paper proposes InferSave, a cost-efficient VM selection framework for cloud based LLM inference. InferSave optimizes KV cache offloading based on Service Level Objectives (SLOs) and workload charac teristics, estimating GPU memory needs, and recommending cost-effective VM instances. Additionally, the Compute Time Calibration Function (CTCF) improves instance selection accuracy by adjusting for discrepancies between theoretical and actual GPU performance. Experiments on AWS GPU instances show that selecting lower-cost instances without KV cache offloading improves cost efficiency by up to 73.7% for online workloads, while KV cache offloading saves up to 20.19% for offline workloads.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。