通过深度转虚拟宽度，让MoE模型更高效扩展。
Mixture of Universal Experts: Scaling Virtual Width via Depth-Width Transformation
- 用通用专家池跨层复用，将深度转化为虚拟宽度。
- 在不同规模下比基线最高提升1.3%，迁移旧模型增益达4.2%。
- 适合追求高效扩展的MoE架构研究者与部署场景。
Mixture-of-Experts (MoE) 将模型容量与单令牌计算量解耦，但其可扩展性仍受限于深度和宽度的物理维度。为此，我们提出 Mixture of Universal Experts (MoUE)，一种 MoE 的泛化形式，引入新缩放维度：虚拟宽度。总体而言，MoUE 旨在跨层复用一个与层无关的通用专家池，在固定每令牌激活预算下，将深度转化为虚拟宽度。然而，仍面临两个挑战：因递归专家复用导致的路由路径爆炸，以及复用带来的暴露程度与传统负载均衡目标之间的不匹配。我们通过三个核心组件解决：分段旋转拓扑实现结构化专家共享，通用专家负载均衡进行深度感知暴露修正，以及轻量级轨迹状态的通用路由器实现连贯多步路由。实验表明，MoUE 在不同缩放范围内始终优于匹配的 MoE 基线，最高提升 1.3%；对现有 MoE 检查点可实现渐进式转换，最高获得 4.2% 提升，并揭示了 MoE 架构的新缩放维度。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) decouples model capacity from per-token computation, yet their scalability remains limited by the physical dimensions of depth and width. To overcome this, we propose Mixture of Universal Experts (MOUE),a MoE generalization introducing a novel scaling dimension: Virtual Width. In general, MoUE aims to reuse a universal layer-agnostic expert pool across layers, converting depth into virtual width under a fixed per-token activation budget. However, two challenges remain: a routing path explosion from recursive expert reuse, and a mismatch between the exposure induced by reuse and the conventional load-balancing objectives. We address these with three core components: a Staggered Rotational Topology for structured expert sharing, a Universal Expert Load Balance for depth-aware exposure correction, and a Universal Router with lightweight trajectory state for coherent multi-step routing. Empirically, MoUE consistently outperforms matched MoE baselines by up to 1.3% across scaling regimes, enables progressive conversion of existing MoE checkpoints with up to 4.2% gains, and reveals a new scaling dimension for MoE architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。