arXiv:2606.15555math.OCcs.AI2026-06被引 1

LLM服务中请求累积内存导致系统自生拥堵,影响吞吐量。

Service-Induced Congestion in Memory-Constrained LLM Serving

论文配图:Service-Induced Congestion in Memory-Constrained LLM Serving
图 1 · 摘自论文原文
  • 构建离散时间动态模型,模拟高并发下内存增长与请求淘汰机制。
  • 同质负载下99%情况会陷入50%吞吐损失的极限环,无法稳定运行。
  • 异构输入长度若互质可避免同步,提升系统稳定性,适合设计调度策略。

在大语言模型(LLM)服务中,每个请求在生成过程中持续积累图形处理单元(GPU)内存,其键值缓存随每生成一个词元而增长。高并发下,内存总量随时间自发增加:服务过程自身制造未来容量压力。当内存超限,系统将活跃请求逐出,丢弃缓存状态并后续重启,造成计算浪费且降低吞吐。本文建立了一个离散时间动力学模型,刻画连续批处理下的请求准入、内存增长与淘汰行为。在饱和输入场景下,系统存在无淘汰的平衡点和含淘汰的极限环。对于同质负载,我们证明无淘汰平衡点不稳定,除测度为零的精确捕获集外,系统均收敛至唯一最差极限环,吞吐损失可达50%。对于异质负载,我们在双类公共输入设置下推导出稳定性判据,并解释生存多项式机制如何推广至多类及异长输入。在输入主导缩放下,互质解码长度可稳定无淘汰平衡点,非互质长度则引发同步模式导致不稳。这些结果揭示了工作负载异质性如何通过去同步化完成来缓解内存受限服务的不稳定性。更广泛地,我们识别出服务诱导拥堵为结构性不稳机制,并提出维持高吞吐的调度设计原则。

原文摘要 · Abstract (English)

In large language model (LLM) serving, each request accumulates persistent graphics processing unit (GPU) memory during service as its key-value cache grows with every generated token. Under high concurrency, aggregate memory usage therefore increases endogenously over time: the service process itself creates future capacity pressure. When memory capacity is exceeded, systems evict active requests, discarding cached state and restarting them later, which wastes computation and reduces throughput. We develop a discrete-time dynamical model of memory-constrained LLM inference that captures admission, memory growth, and eviction under continuous batching. In the saturated-input regime, the system admits both eviction-free fixed points and limit cycles with evictions. For homogeneous workloads, we show that the eviction-free equilibrium is unstable and that, except for a Lebesgue-measure-zero exact-capture set, the system converges to a unique worst-case limit cycle that is asymptotically stable outside this exceptional set, with throughput losses as large as 50%. For heterogeneous workloads, we prove a stability criterion in the two-class common-input setting and explain how the survival-polynomial mechanism generalizes to multiple classes and heterogeneous-input lengths. Under an input-dominated scaling regime, coprime decoding lengths stabilize the eviction-free equilibrium, while non-coprime lengths create synchronized modes that drive instability. These results characterize when workload heterogeneity desynchronizes completions and helps stabilize memory-constrained serving. More broadly, we identify service-induced congestion as a structural instability mechanism and derive scheduling design principles for sustaining high throughput.

LLM服务内存管理系统优化调度策略

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。