通过流量控制提升大模型推理稳定性，确保系统不崩溃。
Flow-Controlled Scheduling for LLM Inference with Provable Stability Guarantees
- 动态控制请求进入速度，避免内存溢出。
- 实验显示吞吐量更高，延迟更低，缓存使用更稳定。
- 适合高并发场景下需要可靠推理的系统设计者。
大语言模型因在众多应用中表现出色而被广泛采用。ChatGPT 和 Gemini 等已服务数亿活跃用户，每日处理数十亿请求，这使优化大模型推理成为关键任务。推理中的主要挑战是解码长度未知，每个请求的内存占用随生成词元增加，可能导致内存溢出并引发系统不稳定。为此，我们提出一种简单的流控框架，通过控制提示词加入活跃请求集的速率来应对该问题。我们推导出任何稳定系统必须满足的必要条件，并建立了在何种条件下我们的算法可证明实现稳定性。实验表明，相比当前常用策略，本方法在保持更高词元和请求吞吐量的同时，显著降低平均与尾部延迟，并实现更稳定的 KV 缓存利用。
原文摘要 · Abstract (English)
Large language models (LLMs) have been widely adopted due to their great performance across a wide range of applications. ChatGPT and Gemini now serve hundreds of millions of active users and handle billions of user requests per day, which puts optimizing LLM inference into the spotlight. A key challenge in LLM inference is that decode lengths are unknown. The memory usage for each request grows with generated tokens, which may lead to overflow and cause system instability. To address this concern, we propose a simple flow-control framework that controls the rate at which prompts join the active set. We derive a necessary condition that any stable system must satisfy and establish sufficient conditions under which our algorithm provably achieves stability. Experiments show that, compared to commonly used strategies in practice, our approach achieves higher token and request throughput, lower average and tail latency, and more stable KV cache utilization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。