通过精简专家中间维度，实现MoE模型高效压缩与推理加速。
Less is MoE: Trimming Experts in Domain-Specialist Language Models
- 基于费舍尔重要性识别关键中间维度，定位任务核心能力分布。
- 仅删减12个中间维度即导致GSM8K准确率崩溃，证明能力集中于特定维度。
- 新方法在50%压缩比下提升吞吐21%，适合部署优化场景。
Mixture-of-Experts（MoE）模型通过条件计算实现强大性能，但其庞大的参数量带来部署挑战。先前的压缩方法在通用基准测试中表现惨败，我们发现原因在于压缩粒度：关键能力分散于专家中，却集中于FFN稀疏中间维度。为此，我们采用费舍尔重要性，优于激活、路由得分和幅度基方法，精准识别出极小的关键维度集合——在Qwen1.5-MoE中，移除仅12个（共135万个）路由-FFN中间维度即导致GSM8K准确率崩塌，而事实知识性能基本保持。基于此，我们提出Fisher-MoE，在FFN内部根据费舍尔重要性移除中间维度。在相同50%压缩率下，该方法保留模型能力，权重内存减少约45%，推理吞吐提升21%。结果表明，中间维度是MoE模型中压缩与排序的有效单元，能力高度集中于此。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) models achieve strong performance through conditional computation, but their large parameter footprint poses deployment challenges. Prior MoE compression approaches catastrophically fail when evaluated on general-purpose benchmarks beyond commonsense reasoning. We trace this failure to the granularity of compression: important capabilities are distributed across experts but concentrated in FFN sparse intermediate dimensions. To identify these dimensions, we use Fisher importance which outperforms activation-, router-score-, and magnitude-based alternatives, and identifies tiny sets of task-critical dimensions: in Qwen1.5-MoE, removing as few as 12 of 1.35M routed-FFN intermediate dimensions collapses GSM8K accuracy while largely preserving factual-knowledge performance. Building on this, we propose Fisher-MoE, which operates within FFN to remove intermediate dimensions ranked by Fisher importance. At the same 50% MoE compression ratio, Fisher-MoE preserves model capability, while reducing weight memory by ~45% and improving inference throughput by 21%. These findings suggest intermediate dimension granularity is an effective unit for both compression and ranking where capability concentrates in MoE models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。