专家自主选择任务，提升模型效率与性能。
Autonomy-of-Experts Models
- 专家自己评估处理能力，按激活强度自选任务
- 700M到4B参数模型均优于传统MoE，效率相当
- 无需路由器，通过低秩分解降低计算开销
混合专家（MoE）模型通常通过路由器将输入分发给特定专家模块，仅激活部分参数，常优于稠密模型。我们指出，路由器决策与专家执行分离是被忽视的关键问题，导致专家选择不佳且学习效率低下。为此，我们提出自治专家（AoE）新范式：专家自主选择处理输入。核心思想是专家自身知晓其处理输入的能力，这种能力体现在内部激活的规模上。AoE移除路由器，专家预先计算输入的内部激活，并依据激活范数排序，仅排名靠前的专家继续前向传播，其余中止。通过低秩权重分解降低预计算开销。这种先自我评估再比对合作的方法，显著改善了专家选择并促进有效学习。我们预训练了700M至4B参数的语言模型，结果表明AoE在保持与传统MoE相当效率的同时表现更优。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) models mostly use a router to assign tokens to specific expert modules, activating only partial parameters and often outperforming dense models. We argue that the separation between the router's decision-making and the experts' execution is a critical yet overlooked issue, leading to suboptimal expert selection and ineffective learning. To address this, we propose Autonomy-of-Experts (AoE), a novel MoE paradigm in which experts autonomously select themselves to process inputs. AoE is based on the insight that an expert is aware of its own capacity to effectively process a token, an awareness reflected in the scale of its internal activations. In AoE, routers are removed; instead, experts pre-compute internal activations for inputs and are ranked based on their activation norms. Only the top-ranking experts proceed with the forward pass, while the others abort. The overhead of pre-computing activations is reduced through a low-rank weight factorization. This self-evaluating-then-partner-comparing approach ensures improved expert selection and effective learning. We pre-train language models having 700M up to 4B parameters, demonstrating that AoE outperforms traditional MoE models with comparable efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。