将调度单位从分词改为分层,显著降低MoE模型推理的延迟和能耗。
From Tokens to Layers: Redefining Stall-Free Scheduling for MoE Serving with Layered Prefill
- 以层组为单位调度,避免重复加载专家权重。
- TTFT降低70%,端到端延迟减少41%,每标记能耗下降22%。
- 适合高并发、低功耗要求的MoE模型部署场景。
大型语言模型在生产环境中的推理需同时满足严格的首令牌时间(TTFT)与令牌间时间(TBT)要求,并在固定计算、内存和互联预算下最大化吞吐量。现代服务系统采用无阻塞调度技术,如分块预填充,将长提示沿令牌维度拆分,并交错执行预填充与解码迭代。尽管能稳定TBT,但该方法在混合专家(MoE)模型中带来显著开销:冗余专家权重加载使内存流量增加高达39%,并提升能耗。本文提出分层预填充,一种以变压器层组为基本调度单元的新范式,专为MoE服务设计。通过垂直划分模型为连续层组,并在层组间交错预填充与解码,分层预填充维持无阻塞解码的同时,消除分块带来的专家权重重载。该方法降低了片外带宽需求,使TTFT最高降低70%,端到端延迟减少41%,每令牌能耗最多下降22%。评估表明,分层预填充在TTFT-TBT帕累托前沿上始终优于分块预填充,减少专家加载流量与能耗,同时保持无阻塞解码。总体而言,将调度轴从令牌转向层,开启了共置环境下高效、节能的MoE服务新范式。
原文摘要 · Abstract (English)
Large Language Model (LLM) inference in production must meet stringent service-level objectives for both time-to-first-token (TTFT) and time-between-token (TBT) while maximizing throughput under fixed compute, memory, and interconnect budgets. Modern serving systems adopt stall-free scheduling techniques such as chunked prefill, which splits the processing of long prompts along the token dimension and interleaves prefill with ongoing decode iterations. While effective at stabilizing TBT, chunked prefill incurs substantial overhead in Mixture-of-Experts (MoE) models: redundant expert weight loads increase memory traffic by up to 39% and inflate energy consumption. We propose layered prefill, a new scheduling paradigm that treats transformer layer groups as the primary scheduling unit, specifically targeting MoE serving. By vertically partitioning the model into contiguous layer groups and interleaving prefill and decode across the groups, layered prefill sustains stall-free decoding while eliminating chunk-induced MoE weight reloads. It reduces off-chip bandwidth demand, lowering TTFT by up to 70%, end-to-end latency by 41% and per-token energy by up to 22%. Evaluations show that layered prefill consistently improves the TTFT--TBT Pareto frontier over chunked prefill, reducing expert-load traffic and energy cost while maintaining stall-free decoding. Overall, shifting the scheduling axis from tokens to layers unlocks a new operating regime for high-efficiency, energy-aware MoE serving in co-located environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。