arXiv:2512.23447cs.CLcs.LG2025-12被引 11

提出轻量辅助损失,让专家与路由决策更匹配,提升MoE模型性能。

Coupling Experts and Routers in Mixture-of-Experts via an Auxiliary Loss

  • 用专家嵌入作代理标记,通过扰动输入强制对齐路由与专家能力。
  • 在30亿到150亿参数模型上训练,显著提升专家专业化程度。
  • 计算开销固定,适合大规模训练,可量化监控专家分工情况。

Mixture-of-Experts(MoE)模型缺乏显式约束来确保路由决策与专家能力对齐,从而限制了模型性能。为此,我们提出专家-路由耦合(ERC)损失,一种轻量级辅助损失,能紧密耦合路由决策与专家能力。该方法将每个专家的路由嵌入视为分配给该专家的令牌的代理标记,并将扰动后的路由嵌入输入专家以获得中间激活。ERC损失施加两个约束:(1) 每个专家对其自身代理标记的激活应高于其他任何专家的代理标记;(2) 每个代理标记应引发对应专家更强的激活,而非其他专家。这些约束共同确保每个路由嵌入真实反映其对应专家的能力,同时每个专家专注于处理实际路由到它的令牌。ERC损失计算效率高,仅需 $n^2$ 次激活,其中 $n$ 为专家数量,开销固定且不随批量大小变化,优于以往随令牌数增长的方法。我们在30亿至150亿参数的MoE-LLM上进行预训练,并在数万亿令牌上进行广泛分析,验证了ERC损失的有效性。此外,该损失提供灵活的专家专业化控制和定量追踪能力,为理解MoE模型训练过程提供重要洞见。

原文摘要 · Abstract (English)

Mixture-of-Experts (MoE) models lack explicit constraints to ensure the router's decisions align well with the experts' capabilities, which ultimately limits model performance. To address this, we propose expert-router coupling (ERC) loss, a lightweight auxiliary loss that tightly couples the router's decisions with expert capabilities. Our approach treats each expert's router embedding as a proxy token for the tokens assigned to that expert, and feeds perturbed router embeddings through the experts to obtain intermediate activations. The ERC loss enforces two constraints on these activations: (1) Each expert must exhibit higher activation for its own proxy token than for the proxy tokens of any other expert. (2) Each proxy token must elicit stronger activation from its corresponding expert than from any other expert. These constraints jointly ensure that each router embedding faithfully represents its corresponding expert's capability, while each expert specializes in processing the tokens actually routed to it. The ERC loss is computationally efficient, operating only on $n^2$ activations, where $n$ is the number of experts. This represents a fixed cost independent of batch size, unlike prior coupling methods that scale with the number of tokens (often millions per batch). Through pre-training MoE-LLMs ranging from 3B to 15B parameters and extensive analysis on trillions of tokens, we demonstrate the effectiveness of the ERC loss. Moreover, the ERC loss offers flexible control and quantitative tracking of expert specialization levels during training, providing valuable insights into MoEs.

MoE专家系统路由优化大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。