arXiv:2604.00421cs.AI2026-04被引 2

不用学习路由模块,直接用隐藏状态选专家,效果不差还更均衡。

Self-Routing: Parameter-Free Expert Routing from Hidden States

  • 从隐藏状态直接提取专家得分,省去专用路由参数。
  • 专家使用更均衡,平均路由熵提高17%,无需负载均衡损失。
  • 在语言和图像任务上表现接近有学习路由的模型,适合追求轻量设计的场景。

Mixture-of-Experts(MoE)层通过每令牌激活少量专家来提升模型容量,通常依赖一个学习得到的路由器将隐藏状态映射为专家分配。本文探讨是否存在必要性:是否必须使用专用学习路由器进行MoE路由?我们提出Self-Routing,一种无参数的路由机制,直接利用令牌隐藏状态中的指定子空间作为专家得分,完全消除路由器投影,同时保持其余MoE结构不变。我们在不同专家数量和模型规模的语言建模任务上评估该方法,并在ImageNet-1K分类任务中与标准学习路由器、随机路由基线及密集非MoE基线进行对比。结果表明,Self-Routing在移除所有专用路由参数的前提下仍能与学习路由器基线竞争,且专家利用率更均衡,平均归一化路由熵约提升17%,且无需显式负载均衡损失。在DeiT-S/16模型上,Self-Routing在ImageNet-1K上的表现也略优于对应的学习路由器MoE。这些发现表明,有效的MoE路由可从隐藏表示本身自然涌现,无需独立学习的路由器模块。

原文摘要 · Abstract (English)

Mixture-of-Experts (MoE) layers increase model capacity by activating only a small subset of experts per token, and typically rely on a learned router to map hidden states to expert assignments. In this work, we ask whether a dedicated learned router is strictly necessary for MoE routing. We propose Self-Routing, a parameter-free routing mechanism that uses a designated subspace of the token hidden state directly as expert logits, eliminating the router projection entirely while leaving the rest of the MoE layer unchanged. We evaluate Self-Routing on language modeling across different expert counts and model scales, and on ImageNet-1K classification by comparing it against a standard learned router, random-routing baselines, and dense non-MoE baselines. Our results show that Self-Routing remains competitive with the learned-router baseline while removing all dedicated routing parameters, and yields more balanced expert utilization, with about 17\% higher average normalized routing entropy and no explicit load-balancing loss. On ImageNet-1K with DeiT-S/16, Self-Routing also slightly improves over the corresponding learned-router MoE. These findings suggest that effective MoE routing can emerge from the hidden representation itself without requiring a separate learned router module.

MoE路由机制无参数专家均衡

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。