arXiv:2605.07260cs.LGcs.CL2026-05被引 5

发现专家模型路由错误频发,尤其在复杂推理时更严重。

When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Models

  • 用反事实分析对比真实路由与同等计算量的替代路由
  • 在困难推理词上,有更低损失的路由却未被选择
  • 仅更新顶层路由网络即可显著提升竞赛题准确率

混合专家(MoE)语言模型将每个词元路由到少数专家,但很少直接评估所选路由是否合理。固定模型后,我们针对同一词元比较标准路由与采样的等计算量替代路由,并以真实推理路径中下一个词的概率作为评分依据。结果表明:在自信词元上,标准路由与路由效用高度一致;但在驱动复杂推理的脆弱词元上,即使存在更低损失的替代路由,也未被选中。该现象在 Qwen3-30B-A3B、GPT-OSS-20B、DeepSeek-V2-Lite 与 OLMoE-1B-7B 中均出现,根源在于标准 top-k 训练仅评估执行路由的损失,且负载均衡依赖整体路由统计。仅更新最终层路由网络(其余冻结),即可显著提升 Qwen3-30B-A3B 与 GPT-OSS-20B 在 AIME 2024+2025 与 HMMT 2025 上的 pass@K,说明部分失败源于路由可及的错配,而非专家容量限制。

原文摘要 · Abstract (English)

Mixture-of-Experts (MoE) language models route each token to a small subset of experts, but whether the routes selected by a trained top-$k$ router are good ones is rarely evaluated directly. Holding the model fixed, we compare each standard route against sampled equal-compute alternatives for the same token and score each by the next-token probability it assigns to the realized token in a verified reasoning trajectory. The result is sharply token-conditional: the standard router is well-aligned with route utility on confident tokens but uninformative on the fragile tokens that drive hard reasoning, where lower-loss equal-compute routes consistently exist inside the frozen model but are not selected. The same pattern holds across Qwen3-30B-A3B, GPT-OSS-20B, DeepSeek-V2-Lite, and OLMoE-1B-7B, and follows structurally from how standard top-$k$ training evaluates routing decisions: the language modeling loss scores only the executed route, and load balancing depends only on aggregate routing statistics. A minimal router-only update to the final-layer router, leaving every expert and every other router frozen, is sufficient to shift pass@K on AIME 2024+2025 and HMMT 2025 for both Qwen3-30B-A3B and GPT-OSS-20B, suggesting that at least part of the failure reflects router-reachable misallocation rather than expert capacity alone.

MoE路由分析推理优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。