arXiv:2604.09780cs.AI2026-04被引 7

MoE专家分工本质是隐状态几何关系,非刻意设计的领域专长。

The Myth of Expert Specialization in MoEs: Why Routing Reflects Geometry, Not Necessarily Domain Expertise

  • 路由器为线性映射,专家使用相似性由隐藏状态相似性决定
  • 五种预训练模型中,令牌与序列层面均验证该规律
  • 专家重叠率仅约60%,提示人类难解读其分工逻辑

混合专家(MoE)在大语言模型中广泛应用,但其“专家专精”机制仍不明确。我们发现,由于MoE路由器为线性映射,隐藏状态相似性既是解释专家使用相似性的必要条件,也是充分条件,因此专精是表示空间的涌现特性,而非路由架构本身所致。我们在五种预训练模型中验证了这一结论,覆盖令牌与序列层级。此外,我们证明负载均衡损失会抑制共享隐藏方向以维持路由多样性,这可能解释了在数据多样性不足(如小批次)时出现的专精崩溃现象。尽管有清晰的机制解释,我们发现预训练MoE中的专精模式难以被人类理解:不同模型对同一问题的回答专家重叠率仅约60%,与完全不同问题相当;提示级路由无法预测展开级路由;深层网络在语义无关输入上表现出近乎相同的专家激活,尤其在推理类模型中。结论是,尽管MoE效率机制已明,但理解专家专精至少与理解大模型隐藏状态几何一样困难,仍是文献中的长期开放问题。

原文摘要 · Abstract (English)

Mixture of Experts (MoEs) are now ubiquitous in large language models, yet the mechanisms behind their "expert specialization" remain poorly understood. We show that, since MoE routers are linear maps, hidden state similarity is both necessary and sufficient to explain expert usage similarity, and specialization is therefore an emergent property of the representation space, not of the routing architecture itself. We confirm this at both token and sequence level across five pre-trained models. We additionally prove that load-balancing loss suppresses shared hidden state directions to maintain routing diversity, which might provide a theoretical explanation for specialization collapse under less diverse data, e.g. small batch. Despite this clean mechanistic account, we find that specialization patterns in pre-trained MoEs resist human interpretation: expert overlap between different models answering the same question is no higher than between entirely different questions ($\sim$60\%); prompt-level routing does not predict rollout-level routing; and deeper layers exhibit near-identical expert activation across semantically unrelated inputs, especially in reasoning models. We conclude that, while the efficiency perspective of MoEs is well understood, understanding expert specialization is at least as hard as understanding LLM hidden state geometry, a long-standing open problem in the literature.

MoE专家系统模型解释性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。