根据输入重要性动态分配专家数量，提升MoE模型性能。
DA-MoE: Towards Dynamic Expert Allocation for Mixture-of-Experts Models
- 用注意力机制衡量令牌重要性，动态决定激活专家数。
- 在GLUE基准上优于现有最先进MoE模型。
- 适合关注模型效率与性能平衡的研究者。
基于Transformer的混合专家（MoE）模型推动了自然语言处理领域的多项技术进步。这些MoE模型通过路由器机制决定为输入令牌激活哪些专家。然而，现有路由器机制对每个令牌固定分配专家数量，忽略了不同输入令牌的重要性差异。本文提出一种新型动态路由器机制——DA-MoE，根据有效的令牌重要性度量动态分配可变数量的专家。首先，我们证明了Transformer注意力机制能自然有效地计算令牌重要性；其次，提出动态路由器，能有效决定最优专家数K，并为每个输入令牌分配前K个专家；最后，在多个基准数据集上的综合实验表明，所提出的DA-MoE方法在流行的GLUE基准上持续优于当前最先进的基于Transformer的MoE模型。
原文摘要 · Abstract (English)
Transformer-based Mixture-of-Experts (MoE) models have been driving several recent technological advancements in Natural Language Processing (NLP). These MoE models adopt a router mechanism to determine which experts to activate for routing input tokens. However, existing router mechanisms allocate a fixed number of experts to each token, which neglects the varying importance of different input tokens. In this study, we propose a novel dynamic router mechanism that Dynamically Allocates a variable number of experts for Mixture-of-Experts (DA-MoE) models based on an effective token importance measure. First, we show that the Transformer attention mechanism provides a natural and effective way of calculating token importance. Second, we propose a dynamic router mechanism that effectively decides the optimal number of experts (K) and allocates the top-K experts for each input token. Third, comprehensive experiments on several benchmark datasets demonstrate that our DA-MoE approach consistently outperforms the state-of-the-art Transformer based MoE model on the popular GLUE benchmark.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。