不重训练就能按需压缩MoE模型，提速近一半且几乎不丢性能。
PreMoE: Proactive Inference for Efficient Mixture-of-Experts
- 用路由器输出预估专家重要性，筛选出高效稀疏版本
- 30B到718B模型上实现50%稀疏度，性能几乎不变
- 适合追求推理效率的部署场景，可定制领域专家或通用模型
Mixture-of-Experts (MoE) 模型具备动态计算能力，但通常以静态全容量形式部署，错失了针对具体应用场景优化的机会。我们提出 PreMoE，一种无需重训练的框架，能主动编译适用于特定部署场景的稀疏 MoE 变体。其核心是预测专家效用（PEU），通过高置信度阈值过滤和对数变换，从路由器输出中稳健估计专家重要性，在极端稀疏条件下仍保持稳定。基于小规模校准集计算的 PEU 分数，PreMoE 能生成面向特定领域的专家排序，从而编译出领域专用模型或高效多领域通用模型，无需任何重新训练。在参数量从 30B 到 718B 的多个 MoE 模型上，PreMoE 实现最高达 50% 的稀疏度，性能几乎无损失。同时揭示出实际部署权衡：专用模型最大化领域内效率，而合成的通用模型则在相同稀疏预算下保留更广的跨领域能力。代码开源：https://github.com/JarvisPei/PreMoE
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) models offer dynamic computation, but are typically deployed as static full-capacity models, missing opportunities for deployment-specific specialization. We introduce PreMoE, a training-free framework that proactively compiles sparse MoE variants for targeted deployment scenarios. At its core is Predicted Expert Utility (PEU), a robust metric for estimating expert importance from router logits through high-confidence threshold filtering and logit transformation, which together stabilize utility estimation under aggressive sparsity. Using PEU scores computed on a small calibration set, PreMoE produces domain-aware expert rankings that can be used to compile either domain-specific specialists or high-efficiency multi-domain generalists, without any retraining. Across MoE models ranging from 30B to 718B parameters, PreMoE achieves up to 50\% sparsity with nearly no performance loss. It further exposes a practical deployment trade-off: specialists maximize in-domain efficiency, while synthesized generalists retain broader cross-domain capability at the same sparsity budget. Code: https://github.com/JarvisPei/PreMoE
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。