解决MoE模型预填充服务中的冗余开销问题,提升推理效率。
MoE-Prefill: Zero Redundancy Overheads in MoE Prefill Serving
- 采用异步专家并行,按权重流式加载专家,避免同步路由开销。
- 在真实与长文本任务上实现1.35-1.59倍吞吐提升,单卡算力利用率超29%。
- 适合大规模预填充场景,如分类、推荐等无需自回归生成的任务。
生产环境中的大模型工作负载越来越多地服务于分类、推荐和验证等判别性任务,其输出仅需一次预填充阶段的logits,无需自回归解码。在混合专家(MoE)模型上服务这类预填充独占任务时,瓶颈并非计算,而是分布式执行带来的内存压力:现有并行策略(张量、专家、流水线并行)以冗余计算、通信和同步为代价换取内存节省,严重降低预填充服务效率。我们发现这些开销源于将专家部署与同步激活路由耦合的设计,这一模式源自解码时代。大批次预填充的长计算前向过程提供了足够宽的每层时间窗口,可后台流式加载专家权重,从而将每层的AllToAll激活通信替换为与计算完全重叠的异步权重AllGather。我们提出MoE-Prefill系统,其后端AsyncEP(异步专家并行)按权重而非激活路由来聚合专家,前端通过前缀感知路由与真实FLOPs负载追踪,强制物理层面的饱和阈值。在Qwen3-235B-A22B模型上,跨四种硬件/精度配置,在真实工作负载下相较最强基线实现1.35-1.37倍吞吐提升,长上下文合成任务最高达1.59倍,同时维持29.8%-36.2%的单卡模型FLOPs利用率。
原文摘要 · Abstract (English)
Production LLM workloads increasingly serve discriminative tasks, such as classification, recommendation, and verification, whose answers are read from the logits of a single prefill pass with no autoregressive decoding. Serving these prefill-only workloads on mixture-of-experts (MoE) models is bottlenecked not by compute but by the distributed execution required to fit the model: existing parallel strategies (tensor, expert, and pipeline parallelism) trade memory pressure for redundant computation, communication, and synchronization, severely degrading MoE prefill serving efficiency. We observe that these overheads stem from coupling expert placement with synchronous activation routing -- a design inherited from the decoding era. The long, compute-bound forward passes of large-batch prefill open a per-layer window wide enough to stream expert weights in the background, replacing per-layer activation AllToAll with asynchronous weight AllGather fully overlapped with computation. We propose MoE-Prefill, a prefill-only serving system whose backend, AsyncEP (Asynchronous Expert Parallelism), gathers experts by weight rather than routing them by activation, and whose frontend co-enforces a physically-derived saturation threshold through prefix-aware routing and true-FLOPs load tracking. On Qwen3-235B-A22B across four hardware/precision configurations, MoE-Prefill delivers 1.35-1.37x throughput over the strongest distributed baseline on real-world workloads and up to 1.59x on long-context synthetic workloads, sustaining 29.8-36.2% per-GPU model FLOPs utilization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。