通过重分配专家原型压缩MoE模型,降低部署内存开销。
ConMoE: Expert-Pool Consolidation via Prototype Reassignment for MoE Compression

- 将原始专家池合并为更小的原型集,固定映射原专家调用。
- 在减少50%专家调用时,深思-16B模型平均性能最优。
- 无需训练,支持局部层内共享,适合资源受限部署场景。
混合专家(MoE)语言模型虽降低每标记计算量,但仍需存储和部署所有专家,导致部署内存开销大。现有后训练压缩方法主要通过剪枝或权重合并来降低该成本。本文将后训练MoE压缩定义为专家池整合:保留一组预训练专家作为可复用原型,并确定性地将每个原始专家引用映射到选定原型。此视角将缩减后的专家池与原专家槽的重用结构解耦,允许在局部层范围内共享原型,同时保持原路由器接口不变。提出ConMoE,一种无需训练的原型重映射框架,通过校准贡献度与可替代性信号选择保留专家,再将原专家调用定向至选定原型,无需权重更新或压缩后微调。在三个预训练MoE语言模型上的实验表明,ConMoE在多种设置下表现优于或媲美强基线,在deepseek-moe-16b-base上实现25%和50%路由专家减少时的最优平均得分,同时在Qwen3-30B-A3B和OLMoE-1B-7B-0125上保持竞争力。消融实验表明,确定性重映射是最稳定的组件,而跨层共享和后处理权重融合则依赖具体模型。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) language models reduce per-token computation but still require storing and serving all experts, making deployment memory-intensive. Existing post-training compression methods mainly shrink this cost by pruning experts or merging their weights. We formulate post-training MoE compression as expert-pool consolidation: retaining a smaller set of pretrained experts as reusable prototypes and deterministically remapping each original expert reference to one selected prototype. This view separates the reduced expert pool from the reuse structure that represents the original expert slots, and allows prototype sharing within local layer scopes while preserving the original router interface. We propose ConMoE, a train-free prototype remapping framework that selects retained experts using calibration-based contribution and replaceability signals, then redirects original expert calls to the selected prototypes without weight updates or post-compression fine-tuning. Experiments on three pretrained MoE language models show that ConMoE matches or outperforms strong pruning and merging baselines in several settings, achieving the best average score on deepseek-moe-16b-base at both 25% and 50% routed-expert reduction, while remaining competitive on Qwen3-30B-A3B and OLMoE-1B-7B-0125. Ablations indicate that deterministic reassignment is the most stable component, whereas broader cross-layer sharing and post-hoc weight fusion are model-dependent.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。