让专家大小不同，让大模型更聪明地分配计算资源。
Mixture of Diverse Size Experts
- 设计不同大小的专家，让每个令牌选最合适的专家生成内容。
- 在多个基准上超越现有MoE，同等参数量下表现更优。
- 提出专家配对策略，均衡多卡负载，适合大规模模型部署。
稀疏激活的混合专家（MoE）架构因其能在不显著增加计算成本的前提下扩展大语言模型而日益流行。然而，当前设计中所有专家大小相同，限制了令牌根据任务需求选择最合适规模专家的能力。本文提出混合多样尺寸专家（MoDSE），在层内引入不同大小的专家。分析表明，在复杂生成任务中，不同尺寸的专家能带来更优预测结果，且专家路由路径在训练后趋于稳定。但多样尺寸可能导致负载不均。为此，我们提出专家对分配策略，实现多GPU间负载均衡。在多个基准上的全面评估显示，MoDSE通过自适应分配参数预算，在保持总参数量和专家数量不变的情况下，显著优于现有MoE架构。
原文摘要 · Abstract (English)
The Sparsely-Activated Mixture-of-Experts (MoE) has gained increasing popularity for scaling up large language models (LLMs) without exploding computational costs. Despite its success, the current design faces a challenge where all experts have the same size, limiting the ability of tokens to choose the experts with the most appropriate size for generating the next token. In this paper, we propose the Mixture of Diverse Size Experts (MoDSE), a new MoE architecture with layers designed to have experts of different sizes. Our analysis of difficult token generation tasks shows that experts of various sizes achieve better predictions, and the routing path of the experts tends to be stable after a training period. However, having experts of diverse sizes can lead to uneven workload distribution. To tackle this limitation, we introduce an expert-pair allocation strategy to evenly distribute the workload across multiple GPUs. Comprehensive evaluations across multiple benchmarks demonstrate the effectiveness of MoDSE, as it outperforms existing MoEs by allocating the parameter budget to experts adaptively while maintaining the same total parameter size and the number of experts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。