让训练好的静态MoE模型变动态，节省一半计算量。
Post-Trained MoE Can Skip Half Experts via Self-Distillation
- 通过自蒸馏加零输出专家，低成本转换静态MoE为动态结构。
- 在11个任务上减少超50%专家计算量，精度损失极小。
- 适合想降低推理成本的模型部署者，尤其大模型场景。
混合专家（MoE）通过稀疏激活高效扩展语言模型，其动态版本能根据输入调整激活专家数量以进一步降低计算量。现有动态MoE方法通常依赖从头预训练或特定任务微调，对已完全训练的MoE模型如何实用化转换研究不足。本文提出零专家自蒸馏适配（ZEDA），一种低成本框架，可将后训练的静态MoE模型转化为高效的动态模型。为稳定结构转换，ZEDA在每层MoE中注入无参数的零输出专家，并通过两阶段自蒸馏进行适配，利用原始MoE作为冻结教师模型，同时引入组级平衡损失。在Qwen3-30B-A3B和GLM-4.7-Flash两个模型上，跨11个涵盖数学、代码和指令遵循的任务验证，ZEDA实现超过50%的专家浮点运算量减少，精度损失微乎其微。相比最强动态MoE基线，分别提升6.1和4.0分，在端到端推理中获得约1.20倍加速。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) scales language models efficiently through sparse expert activation, and its dynamic variant further reduces computation by adjusting the activated experts in an input-dependent manner. Existing dynamic MoE methods usually rely on pre-training from scratch or task-specific adaptation, leaving the practical conversion of fully trained MoE underexplored. Enabling such adaptation would directly alleviate the inference costs by allowing easy tokens to bypass unnecessary expert during serving. This paper introduces Zero-Expert Self-Distillation Adaptation (ZEDA), a low-cost framework that transforms post-trained static MoE models into efficient dynamic ones. To stabilize this architectural conversion, ZEDA injects parameter-free zero-output experts into each MoE layer and adapts the augmented model through two-stage self-distillation, utilizing the original MoE as a frozen teacher and applying a group-level balancing loss. On Qwen3-30B-A3B and GLM-4.7-Flash across 11 benchmarks spanning math, code, and instruction following, ZEDA eliminates over 50% of expert FLOPs at marginal accuracy loss. It outperforms the strongest dynamic MoE baseline by 6.1 and 4.0 points on the two models, and delivers ~1.20$\times$ end-to-end inference speedup.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。