arXiv:2606.24506cs.DCcs.AI2026-06

通过分离权重与缓存,提升冷门MoE模型的显存利用率和长上下文支持能力。

CrossPool: Efficient Multi-LLM Serving for Cold MoE Models through KV-Cache and Weight Disaggregation

论文配图:CrossPool: Efficient Multi-LLM Serving for Cold MoE Models through KV-Cache and Weight Disaggregation
图 1 · 摘自论文原文
  • 将FFN权重与KV缓存分池管理,避免静态权重占用动态缓存空间。
  • 在低并发下仍能高效利用缓存,支持突发长文本请求,尾延迟降低10.4倍。
  • 适合需要高吞吐、长上下文的冷门MoE模型部署场景。

新兴的大语言模型服务越来越多地托管稀疏MoE模型,但多数模型请求稀少,处于冷状态。这带来显存瓶颈:模型权重稳定且由模型决定,而KV缓存是瞬态的且由需求决定。由于冷模型很少同时达到峰值缓存需求,为每个模型预留最坏情况下的缓存容量会浪费内存;共享缓存池可按聚合活跃需求动态分配。然而,当权重与缓存共处同一显存池时,静态权重与动态缓存仍会竞争资源,且在冷负载、低并发下,注意力机制受限于缓存头数量,仅能利用部分复制的缓存容量,导致显存利用率低、长上下文支持弱。本文提出CrossPool,一种面向冷门MoE模型的推理引擎,将前馈网络(FFN)权重与KV缓存分别置于两个显存池中:权重池集中存储多个冷模型的FFN权重,缓存池则动态服务活跃请求,并保持注意力计算局部于缓存。CrossPool结合缓存规划器与虚拟化器、逐层流水调度器以隐藏状态传输延迟,以及持久化内核与控制下推技术,降低CPU-GPU控制开销。通过高效的显存池化,CrossPool支持突发性长上下文请求,显著优于现有基于kvcached的多模型推理系统,尾延迟(P99 TBT)最高降低10.4倍。

原文摘要 · Abstract (English)

Emerging LLM services increasingly host many sparse MoE models, yet most models receive sparse requests and remain cold. This creates a GPU memory problem: model weights are stable and model-determined, while KV-cache is transient and demand-determined. Because cold models rarely reach peak KV-cache demand at the same time, reserving worst-case KV capacity per model wastes memory; a shared KV-cache pool can instead provision aggregate active demand. However, KV-cache sharing is not sufficient when weights and KV-cache remain in a monolithic GPU memory pool. Static weights compete with dynamic KV-cache, and KV-head-limited attention under cold, low-concurrency traffic exposes only a fraction of replicated KV capacity, leading to low GPU memory utilization and weak long-context support. We present CrossPool, a serving engine for cold MoE models that separates FFN weights and KV-cache into two GPU memory pools: a weights pool that consolidates FFN weights across cold models, and a KV-cache pool that dynamically serves active requests while keeping attention local to KV-cache. CrossPool combines a KV-cache planner and virtualizer, a layer-wise pipeline scheduler that hides hidden-state transfers, and persistent kernels with control lowering to reduce CPU-GPU control overhead. With efficient GPU memory pooling, CrossPool underpins bursty long-context requests and outperforms the state-of-the-art kvcached-based multi-LLM serving system, reducing P99 TBT by up to 10.4x.

MoE模型显存优化长上下文多模型服务

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。