首个将计算与显存约束纳入分析的LLM推理稳定性框架
A Queueing-Theoretic Framework for Stability Analysis of LLM Inference with KV Cache Memory Constraints
- 用排队论建模计算与显存双重约束下的推理系统
- 推导出服务稳定与否的严格条件，误差率<10%
- 帮助部署者精准规划集群规模，避免资源浪费或性能崩溃
大型语言模型（LLMs）的快速普及带来了大规模推理的高效挑战。与传统工作负载不同，LLM推理受计算能力和键值（KV）缓存内存开销双重限制，后者虽加速解码但迅速耗尽GPU内存。本文提出首个将计算和GPU内存约束纳入分析的排队论框架。基于该框架，我们推导出决定推理服务能否持续应对请求而不会导致队列无限增长的稳定与不稳定条件。这一结果为系统部署提供了强大工具，可解决核心的GPU资源配置难题。结合估算的请求到达率与推导出的稳定服务速率，运维人员可计算出避免过度采购和性能违规的必要集群规模。我们在真实GPU生产环境中通过大量实验验证了理论预测，结果显示预测稳定性条件高度准确，偏差通常在10%以内。
原文摘要 · Abstract (English)
The rapid adoption of large language models (LLMs) has created significant challenges for efficient inference at scale. Unlike traditional workloads, LLM inference is constrained by both computation and the memory overhead of key-value (KV) caching, which accelerates decoding but quickly exhausts GPU memory. In this paper, we introduce the first queueing-theoretic framework that explicitly incorporates both computation and GPU memory constraints into the analysis of LLM inference. Based on this framework, we derive rigorous stability and instability conditions that determine whether an LLM inference service can sustain incoming demand without unbounded queue growth. This result offers a powerful tool for system deployment, potentially addressing the core challenge of GPU provisioning. By combining an estimated request arrival rate with our derived stable service rate, operators can calculate the necessary cluster size to avoid both costly over-purchasing and performance-violating under-provisioning. We further validate our theoretical predictions through extensive experiments in real GPU production environments. Our results show that the predicted stability conditions are highly accurate, with deviations typically within 10%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。