arXiv:2609.04453cs.LGcs.AI2026-09

过分散路由下专家剪枝会失效,需新方法避免领域性能退化。

When Load-Balancing Goes Too Far: Expert Pruning in Over-Dispersed Mixture-of-Experts Models

论文配图:When Load-Balancing Goes Too Far: Expert Pruning in Over-Dispersed Mixture-of-Experts Models
图 1 · 摘自论文原文
  • 提出基于领域感知的最小化最差退化评分法(MESA),动态优化专家保留策略。
  • 在25%剪枝率下,MESA在11个基准中7个表现优于基线,且内存开销更低。
  • 发现困惑度无法预测任务准确率,激活与频率评分各有优劣,需权衡选择。

专家剪枝通过移除路由器判定为低重要性的专家来降低混合专家(MoE)模型的内存和推理成本,通常假设路由器概率能可靠反映重要性。然而我们发现,在训练中采用激进负载均衡导致的过分散路由下,该假设失效:令牌几乎均匀分布于各专家,重要性信号崩溃。此时困惑度不再预测下游任务准确率——在gpt-oss-20B上,最低困惑度剪枝配置数学推理能力最差,最高困惑度配置则保持较好表现。而标准路由(如Mixtral-8x7B-Instruct)下,困惑度与准确率同步下降。过分散路由下的剪枝暴露了能力权衡:激活感知评分保数学推理但严重损害知识密集型科学任务(GPQA下降18分),频率评分则相反。我们提出最小化最差退化评分分配(MESA),一种领域感知迭代方法,持续提升当前最差领域所服务专家的重要性分数,以最小化最坏情况下的性能退化。在25%专家剪枝率下,MESA实现跨领域的最小最差退化,11个基准中有7个优于激活感知基线,同时显著降低内存占用,并可推广至gpt-oss-120B、Gemma-4-26B-A4B和OLMoE-1B-7B。

原文摘要 · Abstract (English)

Expert pruning reduces the memory and serving cost of Mixture-of-Experts (MoE) models by removing low-importance experts identified by the router, assuming router probabilities provide a reliable importance signal. We observe that this assumption breaks down under over-dispersed routing, a regime associated with aggressive load-balancing during training, in which tokens are distributed nearly uniformly across experts and importance signals collapse. In this regime, perplexity does not predict downstream task accuracy: on gpt-oss-20B, the lowest-perplexity pruning configuration yields the worst mathematical reasoning, while the highest-perplexity configuration preserves it. This does not occur under standard routing (e.g., Mixtral-8x7B-Instruct), where perplexity and accuracy degrade together. Pruning under over-dispersed routing also exposes a capability trade-off in which no single scoring metric dominates: activation-aware scoring preserves mathematical reasoning but severely degrades knowledge-intensive science (an 18-point gap on GPQA), whereas frequency-based scoring exhibits the reverse. We propose Minimax Expert Score Allocation (MESA), a domain-aware method that iteratively boosts importance scores for experts serving whichever domain is currently worst-affected, minimizing worst-case domain degradation rather than average accuracy. At 25% expert pruning MESA achieves the smallest worst-case degradation across domains, outperforming activation-aware baselines on 7 of 11 benchmarks at a correspondingly reduced memory footprint, and it generalizes to gpt-oss-120B, Gemma-4-26B-A4B, and OLMoE-1B-7B. Our results indicate that over-dispersed routing is a qualitatively distinct pruning regime in which standard assumptions fail, and that recognizing it is a prerequisite for principled expert pruning of load-balanced MoE models.

专家模型剪枝负载均衡多任务

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。