arXiv:2604.17837cs.AIcs.CL2026-04被引 2

揭示MoE模型中专家路径的语义专一性,突破传统专家解释局限。

Polysemantic Experts, Monosemantic Paths: Routing as Control in MoEs

论文配图:Polysemantic Experts, Monosemantic Paths: Routing as Control in MoEs
图 1 · 摘自论文原文
  • 将隐藏状态分解为控制信号与内容通道,实现路由机制解耦
  • 不同标点符号依上下文走不同语义路径,体现动态功能分化
  • 轨迹比专家更适合作为可解释性单位,适用于模型分析与调试

大型语言模型的残差流既是状态也是指令:它编码当前上下文并决定下一步变换。我们提出一种无需参数的Mixture-of-Experts模型分解方法,将每一层的隐藏状态拆分为一个因果驱动路由的控制信号,以及一个对路由器不可见的内容通道。在六种MoE架构中,内容通道保留了语言、词元身份、位置等表层特征,而控制信号则编码随层变化的抽象函数。由于每次路由决策带宽低,这种传递机制迫使各层形成组合式专精。尽管单个专家仍具多义性,但专家路径呈现单义性,按语义功能聚类,跨越语言与形式。同一个词元(如“:”)在作为类型标注、引导冒号或时间分隔符时,会遵循不同的语义轨迹。该分解揭示了结构来源:控制子空间中的聚类比完整表示更具单义性。因此,MoE中自然的可解释单元并非专家,而是轨迹。

原文摘要 · Abstract (English)

An LLM's residual stream is both state and instruction: it encodes the current context and determines the next transformation. We introduce a parameter-free decomposition for Mixture-of-Experts models that splits each layer's hidden state into a control signal that causally drives routing and an orthogonal content channel invisible to the router. Across six MoE architectures, we find that models preserve surface-level features (language, token identity, position) in the content channel, while the control signal encodes an abstract function that rotates from layer to layer. Because each routing decision is low-bandwidth, this hand-off forces compositional specialization across layers. While individual experts remain polysemantic, expert paths become monosemantic, clustering tokens by semantic function across languages and surface forms. The same token (e.g., ":") follows distinct trajectories depending on whether it serves as a type annotation, an introductory colon, or a time separator. Our decomposition identifies the source of this structure: clusters in the control subspace are substantially more monosemantic than those in the full representation. As a result, the natural unit of interpretability in MoEs is not the expert but the trajectory.

MoE可解释性路径追踪语义专一

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。