arXiv:2608.08650cs.CL2026-08

解析专家混合模型的架构演进,揭示路由与并行的深层设计逻辑。

The Evolution of Mixture-of-Experts Architectures in Large Language Models: Routing, Topology, Load Balancing, and Expert Parallelism

  • 按五个维度重构专家系统:粒度、拓扑、路由自由度等
  • 提出四维控制平面,统一算法与系统设计关系
  • 适合研究大模型架构或系统优化的读者

专家混合模型在保持每标记计算量受限的同时提升参数容量,但其架构演化不能仅靠模型发布顺序来解释。本技术综述整合了主要论文、官方技术报告及先前综述,从五个耦合维度组织现代专家混合系统:专家粒度、拓扑结构、路由自由度、负载均衡范围和执行结构。我们以依赖图形式呈现八个架构里程碑,包含六条主线发展与两条正交分支,而非八代连续演进。随后通过四个控制平面分析各系统:专家拓扑、路由、负载平衡与专家并行。这些平面分别定义专家存在方式、每个标记处理的专家、整体负载控制机制以及计算映射到物理设备的方式。该框架将Top-k路由、共享专家、细粒度专家、动态专家组合等算法选择,与标记分发、设备放置、全对全通信、通信-计算重叠等系统问题相连接。最后通过等预算预训练实验、质量与系统指标,总结开放研究问题。核心趋势是从单纯激活更多稀疏参数,转向解耦语义路由、计算预算与物理执行。

原文摘要 · Abstract (English)

Mixture-of-Experts models increase parameter capacity while keeping the computation activated by each token bounded, but their architectural evolution cannot be explained by a chronological list of model releases alone. This technical survey synthesizes primary papers, official technical reports, and prior surveys to organize modern Mixture-of-Experts systems along five coupled dimensions: expert granularity, expert topology, routing freedom, the scope of load balancing, and execution structure. We describe eight architectural milestones as a dependency graph with six mainline developments and two orthogonal branches, rather than as eight successive generations. We then analyze individual systems through four control planes: Expert Topology, Routing, Balance, and Expert Parallelism. These planes specify which experts exist, which experts process each token, how aggregate load is controlled, and how selected computation is mapped onto physical devices. The framework connects algorithmic choices such as Top-k routing, shared experts, fine-grained experts, and dynamic expert composition with systems concerns including token dispatch, device placement, all-to-all communication, and communication-computation overlap. We conclude with equal-budget pretraining experiments, quality and systems metrics, and open research questions. The main trend is a shift from merely activating more sparse parameters toward decoupling semantic routing, computational budgets, and physical execution.

专家混合大模型架构系统优化路由机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。