将专家模型权重按需加载，显著提升大模型推理速度。
FluxMoE: Decoupling Expert Residency for High-Performance MoE Serving
- 专家权重不常驻显存，按需加载并使用后立即释放。
- 在内存受限场景下，推理吞吐量最高提升3倍。
- 适合部署超大规模专家模型的高效服务系统。
混合专家（MoE）模型已成为扩展大型语言模型的主要范式，但其参数规模快速膨胀导致推理时存在根本性低效：多数专家权重在GPU内存中闲置，与关键的键值（KV）缓存争抢资源。由于KV缓存容量直接影响服务吞吐量，这种内存分配失衡造成显存利用率低下和性能下降。本文提出FluxMoE，一种新型MoE推理系统，通过解耦专家参数与持久显存驻留。FluxMoE引入专家分页抽象，将专家权重视为可流式传输的临时资源，按需加载并在使用后立即卸载，从而优先保障吞吐量关键的运行时状态占用显存。我们在vLLM基础上实现FluxMoE，支持在严重内存约束下的高效MoE推理。实验表明，在内存密集型场景下，FluxMoE相比vLLM最高可实现3.0×的吞吐量提升，且不影响模型精度。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) models have become a dominant paradigm for scaling large language models, but their rapidly growing parameter sizes introduce a fundamental inefficiency during inference: most expert weights remain idle in GPU memory while competing with performance-critical runtime state such as the key-value (KV) cache. Since KV cache capacity directly determines serving throughput, this mismatch leads to underutilized memory and degraded performance. In this paper, we present FluxMoE, a new MoE inference system that decouples expert parameters from persistent GPU residency. FluxMoE introduces an expert paging abstraction that treats expert weights as streamed, transient resources, materializing them on demand and evicting them immediately after use, allowing GPU memory to be preferentially allocated to throughput-critical runtime state. We implement FluxMoE atop vLLM to enable efficient MoE inference under severe memory constraints. Experimental results demonstrate that FluxMoE achieves up to 3.0$\times$ throughput gains over vLLM in memory-intensive regimes, without compromising model fidelity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。