不重训练也能压缩MoE模型，只需轻量级路由校准即可恢复性能。
Is Retraining-Free Enough? The Necessity of Router Calibration for Efficient MoE Compression
- 只更新路由模块参数，不改专家参数，实现免重训练压缩。
- 在细粒度MoE上性能提升显著，因路由边界更复杂。
- 适合部署时需快速压缩模型且无法重训练的场景。
Mixture-of-Experts (MoE) 模型能高效扩展容量，但其庞大的参数量导致部署时内存瓶颈。我们把免重训练的MoE压缩分为三类：专家剪枝、专家编辑和专家合并，并发现压缩后性能下降主要源于一个被忽视的因素：专家变化而路由未更新导致的路由-专家不匹配。我们认为有效的免重训练压缩应保持专家参数不变，仅允许轻量级路由校准。为此，我们提出路由器知识蒸馏（Router KD），通过在无标签校准数据上蒸馏原始模型的下一个词分布，仅更新极小比例的参数（即路由模块）。在所有三类代表性方法上的实验均显示性能一致恢复，尤其在细粒度MoE（多个小型专家）中提升更大，因其路由决策边界更复杂。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) models scale capacity efficiently, but their massive parameter footprint creates a deployment-time memory bottleneck. We organize retraining-free MoE compression into three paradigms - Expert Pruning, Expert Editing, and Expert Merging - and show that persistent post-compression degradation largely stems from a neglected factor: router-expert mismatch when experts are changed but the router is left untouched. We argue that effective retraining-free compression should avoid updating expert parameters while allowing lightweight router calibration. To this end, we propose Router Knowledge Distillation (Router KD), which updates only a tiny fraction of parameters (the router) by distilling the original model's next-token distribution on unlabeled calibration data. Experiments across representative methods in all three paradigms demonstrate consistent performance recovery, with substantially larger gains in fine-grained MoEs (many small experts) than in coarse-grained MoEs due to their more complex routing decision boundaries.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。