动态调整专家激活数量与分配,让模型更聪明地计算。
DynaMoE: Dynamic Token-Level Expert Activation with Layer-Wise Adaptive Capacity for Mixture-of-Experts Neural Networks
- 按输入复杂度动态决定每令牌激活专家数
- 不同任务和规模下最优专家分配策略不同
- 显著提升参数效率,训练更稳定
Mixture-of-Experts(MoE)架构在保持计算效率的同时实现了神经网络的高效扩展。然而,传统MoE依赖两个固定设计:(1) 固定的Top-K路由,每令牌恰好激活K个专家;(2) 所有层间专家分配均匀。本文提出DynaMoE,通过动态令牌级专家激活和逐层自适应容量分配,打破上述限制。DynaMoE引入一种基于输入复杂度动态变化活跃专家数的路由机制,并实现六种专家容量调度策略,包括递减、递增、金字塔和波浪模式。我们理论分析了动态路由带来的表达能力提升,并推导出计算效率边界。在MNIST、Fashion-MNIST、CIFAR-10(图像分类)及Recycling-the-Web(语言建模)数据集上,多尺度模型实验表明,相比静态基线,DynaMoE实现更优的参数效率。关键发现:最优专家调度具有任务与规模依赖性——图像分类中递减调度(早期层集中容量)优于均匀基线;语言建模中,小型模型适用递减,中型用均匀,大型用递增。此外,动态路由降低训练梯度方差,提升收敛稳定性。DynaMoE为神经网络自适应计算提供了新框架,为MoE架构设计提供原则性指导。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) architectures have emerged as a powerful paradigm for scaling neural networks while maintaining computational efficiency. However, standard MoE implementations rely on two rigid design assumptions: (1) fixed Top-K routing where exactly K experts are activated per token, and (2) uniform expert allocation across all layers. This paper introduces DynaMoE, a novel MoE framework that relaxes both constraints through dynamic token-level expert activation and layer-wise adaptive capacity allocation. DynaMoE introduces a principled routing mechanism where the number of active experts per token varies based on input complexity. Concurrently, the framework implements six distinct scheduling strategies for distributing expert capacity across network depth, including descending, ascending, pyramid, and wave patterns. We theoretically analyze the expressivity gains of dynamic routing and derive bounds on computational efficiency. Through extensive experiments on MNIST, Fashion-MNIST, CIFAR-10 (image classification), and Recycling-the-Web (language modeling) across multiple model scales, we demonstrate that DynaMoE achieves superior parameter efficiency compared to static baselines. Our key finding is that optimal expert schedules are task- and scale-dependent: descending schedules (concentrating capacity in early layers) outperform uniform baselines on image classification. For language modeling, optimal schedules vary by model size, descending for Tiny, ascending for Small, and uniform for Medium. Furthermore, dynamic routing reduces gradient variance during training, leading to improved convergence stability. DynaMoE establishes a new framework for adaptive computation in neural networks, providing principled guidance for MoE architecture design.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。