提出分层专家路由机制，解决稀疏专家模型的负载不均与路由崩溃问题。
Hierarchical Mixture-of-Experts with Two-Stage Optimization
- 分两阶段控制路由：组间均衡与组内专精协同
- 7B模型在580亿词上降低5.6%困惑度，专家负载提升40%
- 适合大规模多模态预训练，对模型规模和专家数量鲁棒
稀疏专家混合（MoE）模型通过将每个标记路由到少量专家来扩展容量。然而，其路由器存在根本性权衡：良好的负载均衡会抑制专家专长，而过度追求多样性常导致路由崩溃。我们提出Hi-MoE，一种分组式MoE框架，将路由控制分解为两个耦合层级：(i) 组间平衡，确保专家组间流量公平；(ii) 组内专精，促进互补专家行为并防止组内崩溃。分析表明，分层目标重塑了路由器，从而实现稳定专长并缓解崩溃。在NLP与视觉基准上，性能持续优于近期稀疏路由与分组MoE基线，并通过缩放研究（模型大小、专家数量）与针对性消融验证鲁棒性。在580亿词大规模预训练中，Hi-MoE-7B相较OLMoE-7B实现5.6%困惑度下降与40%专家平衡提升，覆盖多样评估领域。
原文摘要 · Abstract (English)
Sparse Mixture-of-Experts (MoE) models scale capacity by routing each token to a small subset of experts. However, their routers exhibit a fundamental trade-off: strong load balancing can suppress expert specialization, while aggressive diversity often causes routing collapse. We propose Hi-MoE, a grouped MoE framework that decomposes routing control into two coupled levels: (i) inter-group balancing that enforces fair traffic across expert groups, and (ii) intra-group specialization that promotes complementary expert behaviors while preventing within-group collapse. Our analysis provides a principled explanation of how our hierarchical objectives reshape the router, thereby promoting stable specialization and mitigating collapse. We observe consistent improvements over recent sparse-routing and grouped-MoE baselines across NLP and vision benchmarks, and confirm robustness via scaling studies (model size, expert count) and targeted ablations. In large-scale pre-training on 58B tokens, Hi-MoE-7B achieves a 5.6% perplexity reduction and a 40% improvement in expert balance over OLMoE-7B across diverse evaluation domains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。