将专家混合模型转化为密集型模型，解决内存占用难题
Pruning and Distilling Mixture-of-Experts into Dense Language Models
- 通过评分、分组和拼接专家，生成密集前馈网络
- 在相同参数量下，性能比纯密集模型高6.3个百分点
- 适合资源受限场景的模型部署，尤其关注推理效率
专家混合（MoE）是当前前沿语言模型的主流架构，但需加载全部专家参数，不适用于内存受限的部署场景。现有压缩方法虽减少专家数量，但输出仍为MoE结构，存在相同局限。本文提出首个系统性框架，将训练好的MoE模型转换为标准全连接密集架构：对专家进行评分、选择与分组，随后拼接为密集前馈网络，并通过知识蒸馏从MoE教师模型中微调。我们在Qwen3-30B-A3B上评估了7种评分、5种分组和2种幅度缩放方法，共350种配置。结果表明，评分方法影响最大，新提出的多样性感知评分在Qwen3-30B-A3B、DeepSeek-V2-Lite和GPT-OSS-20B上均优于已有方法。在参数量匹配条件下，经约40亿词元蒸馏后，MoE转密集模型在下游平均准确率上比密集剪枝模型高出6.3个百分点，且训练速度提升1.6倍。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) is now the dominant architecture for frontier language models, yet it requires all expert parameters to be loaded in memory, making it less preferable for memory-constrained deployment. Existing compression methods reduce the number of experts but the output remains an MoE model with the same fundamental limitation. We present the first systematic framework for converting a trained MoE into a standard fully dense architecture: experts are scored, selected, and grouped, then concatenated into a dense FFN and refined by knowledge distillation from the MoE teacher. We evaluate 7 scoring, 5 grouping, and 2 magnitude scaling methods across a range of selected expert counts on Qwen3-30B-A3B, yielding 350 configurations. We find that the choice of scoring method is the most impactful, with our novel diversity-aware scoring consistently outperforming prior methods on Qwen3-30B-A3B, DeepSeek-V2-Lite, and GPT-OSS-20B. Under a controlled comparison at matched parameter count, MoE-to-dense outperforms dense-to-dense pruning by +6.3 pp in average downstream accuracy after ~4B-token distillation at 1.6x faster training wall-clock speed.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。