ExFold统一加速MoE模型预填充与解码,无需训练即可提升速度。
ExFold: Unified Expert Folding for Training-Free MoE Prefill-Decode Acceleration

- 通过统一框架在两类推理阶段折叠冗余专家,仅保留关键专家计算
- 实现最高1.41倍的首次令牌延迟降低和2.45倍的每令牌处理速度提升
- 适用于追求低延迟部署的MoE模型,尤其适合vLLM生态
混合专家(MoE)模型通过稀疏激活扩展容量并控制单标记计算量。然而,低延迟服务日益困难,因预填充阶段受标记级专家计算主导,而解码阶段受限于批量激活专家集带来的内存访问压力。现有免训练加速方法仅优化单一资源代理,或忽略被排除专家的贡献,或仅隐式近似。本文提出ExFold,一种统一的免训练专家折叠框架,可联合加速MoE的预填充与解码。ExFold将两个阶段均视为受预算约束的输出近似问题:仅执行特定阶段的受限专家集,并利用校准的标量投影器将被剔除专家的贡献映射到保留专家上。基于大量专家输出方向对齐但幅度不同的观察,ExFold在无标签数据上校准成对标量投影矩阵,并在推理时使用以折叠被排除专家的贡献。在此视角下,预填充加速变为标记级Top-K折叠,解码加速变为批量级专家池折叠。两阶段仅在保留专家选择方式上不同,而被排除贡献均由同一折叠机制恢复。我们在vLLM中实现ExFold为即插即用插件,配备轻量级专家折叠CUDA内核,在保持约99%原始平均质量的同时,带来最高1.41倍的TTFT和2.45倍的TPOT加速。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) models scale capacity for strong quality while keeping per-token compute bounded through sparse expert activation. Yet low-latency MoE serving is increasingly challenging, because it spans two inference phases with fundamentally different bottlenecks: prefill is dominated by token-wise expert computation, whereas decode is constrained by memory traffic from the batch-wise activated expert set. However, existing training-free acceleration methods optimize only a single resource proxy, either the experts each token executes or the experts a batch activates, and either discard the excluded experts' contribution or leave it only implicitly approximated. In this paper, we propose ExFold, a unified training-free expert-folding framework for jointly accelerating MoE prefill and decode. ExFold casts both prefill and decode as one budgeted output-approximation problem: execute only a phase-specific constrained expert set while projecting the contribution of budget-excluded experts onto retained experts using calibrated scalar projectors. Motivated by the observation that many expert outputs are directionally aligned but differ in magnitude, ExFold calibrates a pairwise scalar-projector matrix on unlabeled data and uses it at inference time to fold excluded expert contributions into retained experts. Under this view, prefill acceleration becomes token-level Top-K folding, and decode acceleration becomes batch-level expert-pool folding. The two phases differ only in how retained experts are selected, while excluded contributions are recovered by one shared folding mechanism. We implement ExFold as a plug-and-play plugin in vLLM, with a lightweight expert-folding CUDA kernel, delivering up to 1.41x TTFT and 2.45x TPOT speedups while retaining about 99% of the original average quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。