arXiv:2608.15299cs.LGcs.AI2026-08

让专家模型各层灵活分配专家数量,提升性能与效率。

MAPLE: MoE Adaptive Plug-and-play Layer-wise Expert allocation

论文配图:MAPLE: MoE Adaptive Plug-and-play Layer-wise Expert allocation
图 1 · 摘自论文原文
  • 根据各层敏感度动态分配专家数,无需重训练。
  • 仅用75%专家量,准确率反而超过原100%设置。
  • 适合追求高效推理的MoE模型部署者。

稀疏激活的专家混合(MoE)Transformer通常在所有层中固定相同数量的路由专家,这一做法忽视了层间冗余性的显著差异。我们证明这种均匀性系统性地次优,并提出MAPLE——一种即插即用框架,可在不修改权重或重新训练的前提下,对任何预训练MoE大模型实现层间专家预算的非均匀再分配。核心贡献是基于敏感度的闭式分配:通过探测各层对专家数量变化的响应,使用三种度量量化敏感度,推导出解析最优的预算分配方案,将容量导向敏感层,减少冗余层的资源。该闭式解进一步通过敏感度约束的遗传搜索优化,利用层敏感度作为先验引导探索,实现更快收敛与更优分配质量。在四种不同规模与架构的MoE模型上,MAPLE在75%路由专家预算下优于均匀与剪枝基线。特别地,在DeepSeek-MoE-16B上,仅使用75%专家却在ARC-E、ARC-C和BoolQ上分别将准确率从65.09、48.49、80.03提升至71.40、51.50、82.38。这些性能提升转化为实际部署效率:在SGLang中实现单卡端到端服务延迟降低32.2%,吞吐量提升47.4%。结果表明,精心设计的异构分配比简单增加专家数量更有效,确立其为提升MoE效率的原理性且实用路径。

原文摘要 · Abstract (English)

Sparsely-activated Mixture-of-Experts (MoE) Transformers universally fix the same number of routed experts across all layers, a convention that ignores the well-documented heterogeneity in layer-wise redundancy. We demonstrate that this uniformity is systematically suboptimal and propose MAPLE, a plug-and-play framework that reallocates the routed-expert budget heterogeneously across layers of any pretrained MoE LLM, without modifying weights or requiring retraining. Our core contribution is a closed-form sensitivity-guided allocation: we probe each layer's response to variation in expert count, quantify sensitivity using three measures, and derive an analytically optimal budget assignment that directs capacity towards sensitive layers and absorbs reductions in redundant layers. This closed-form solution is further refined by a sensitivity-constrained genetic search that uses layer-wise sensitivity as a prior to guide exploration, yielding faster convergence and superior allocation quality. On four MoE models spanning different scales and architectures, MAPLE outperforms uniform and pruning-based baselines under a 75% routed-expert budget. Notably, on DeepSeek-MoE-16B, MAPLE uses only 75% of the experts yet surpasses the original 100% expert-uniform baseline on ARC-E, ARC-C, and BoolQ, improving accuracy from 65.09 to 71.40, 48.49 to 51.50, and 80.03 to 82.38, respectively. These accuracy gains translate into measured deployment efficiency: implementing MAPLE in SGLang reduces single-GPU end-to-end serving latency by 32.2% and improves throughput by 47.4%. These results show that well-designed heterogeneous allocation can be more effective than simply activating more experts, establishing it as a principled and practical axis for improving MoE efficiency.

MoE专家分配模型效率即插即用

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。