发现专家模型中存在恒定的中心计算组,打破领域专精假象。
The Illusion of Specialization: Unveiling the Domain-Invariant "Standing Committee" in Mixture-of-Experts Models
- 通过分组分析路由行为,揭示跨领域稳定的专家核心团队。
- 核心团队占多数路由流量,即使有共享专家也持续主导计算。
- 适合关注模型结构偏见与训练优化的研究者阅读。
Mixture of Experts 模型普遍被认为通过稀疏路由实现领域专精。本文提出后验分析框架 COMMITTEE AUDIT,从专家组层面而非单个专家分析路由行为。在三个代表性模型和 MMLU 基准上,我们发现一个领域不变的「核心委员会」:一组被持续调用的专家,在不同领域、层和路由预算下始终占据主要路由质量。定性分析表明,核心委员会锚定推理结构与语法,外围专家负责领域特定知识。该发现揭示了向中心化计算的强烈结构性偏见,说明专家模型中的专精远不如普遍认为的那么普遍。这种内在偏见意味着当前负载均衡等训练目标可能违背模型自然优化路径,限制训练效率与性能。
原文摘要 · Abstract (English)
Mixture of Experts models are widely assumed to achieve domain specialization through sparse routing. In this work, we question this assumption by introducing COMMITTEEAUDIT, a post hoc framework that analyzes routing behavior at the level of expert groups rather than individual experts. Across three representative models and the MMLU benchmark, we uncover a domain-invariant Standing Committee. This is a compact coalition of routed experts that consistently captures the majority of routing mass across domains, layers, and routing budgets, even when architectures already include shared experts. Qualitative analysis further shows that Standing Committees anchor reasoning structure and syntax, while peripheral experts handle domain-specific knowledge. These findings reveal a strong structural bias toward centralized computation, suggesting that specialization in Mixture of Experts models is far less pervasive than commonly believed. This inherent bias also indicates that current training objectives, such as load-balancing losses that enforce uniform expert utilization, may be working against the model's natural optimization path, thereby limiting training efficiency and performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。