动态调整专家选择比例,让模型更聪明地分配计算资源。
DTop-p MoE: Sparsity-Controlled Dynamic Top-p MoE for Foundation Model Pre-training
- 用自适应阈值动态控制每次路由选多少专家
- 在保持计算量与Top-k相当的前提下提升性能
- 适合大模型训练,尤其对计算资源敏感的场景
稀疏混合专家(MoE)架构是高效扩展模型容量的关键,但标准的Top-$k$路由机制固定了稀疏模式,未能考虑标记难度和层间计算需求的差异。Top-$p$路由通过累计概率阈值动态选择专家,更具适应性。然而,现有固定全局阈值的实现仅带来微弱改进,对超参数敏感且计算成本不可控。本文提出DTop-$p$,一种可调节稀疏度的动态路由机制:通过比例-积分控制器学习动态的Top-$p$概率阈值,并采用动态归一化支持各层专家选择,在全局稀疏性约束下实现灵活调度。在大语言模型和扩散变换器上的实验表明,DTop-$p$始终优于Top-$k$和固定Top-$p$基线,同时平均FLOPs与Top-$k$相当。分析证实其在专家粒度、总专家容量、模型规模和数据集大小上均具良好扩展性,为大模型预训练提供了鲁棒高效的MoE框架。
原文摘要 · Abstract (English)
Sparse Mixture-of-Experts architectures are essential for scaling model capacity efficiently, yet the standard Top-$k$ routing imposes a rigid sparsity pattern that ignores the intrinsic variance in token difficulty and layer-specific computational needs. Top-$p$ routing is more adaptive because it selects experts until their cumulative routing probability reaches a threshold, allowing confident tokens to use fewer experts and ambiguous tokens to recruit more. However, we demonstrate that existing naive Top-$p$ implementations with fixed global probability thresholds provide only marginal gains over Top-$k$, suffer from hyperparameter sensitivity, and result in uncontrolled computational costs. In this paper, we propose **DTop-$p$**, a sparsity-controllable dynamic routing mechanism that learns the Top-$p$ probability threshold with a Proportional-Integral controller and uses dynamic routing normalization to support layer-wise expert selection under a global sparsity constraint. Extensive experiments on Large Language Models and Diffusion Transformers demonstrate that **DTop-$p$** consistently outperforms both Top-$k$ and fixed Top-$p$ baselines while matching the average FLOPs of Top-$k$ MoE. Our analysis confirms that **DTop-$p$** exhibits strong scaling properties across expert granularity, total expert capacity, model size, and dataset size, offering a robust and efficient MoE framework for foundation model pre-training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。