揭示MoE模型中专家路径的语义专一性,突破传统专家解释局限。
Polysemantic Experts, Monosemantic Paths: Routing as Control in MoEs

- 将隐藏状态分解为控制信号与内容通道,实现路由机制解耦
- 不同标点符号依上下文走不同语义路径,体现动态功能分化
- 轨迹比专家更适合作为可解释性单位,适用于模型分析与调试
大型语言模型的残差流既是状态也是指令:它编码当前上下文并决定下一步变换。我们提出一种无需参数的Mixture-of-Experts模型分解方法,将每一层的隐藏状态拆分为一个因果驱动路由的控制信号,以及一个对路由器不可见的内容通道。在六种MoE架构中,内容通道保留了语言、词元身份、位置等表层特征,而控制信号则编码随层变化的抽象函数。由于每次路由决策带宽低,这种传递机制迫使各层形成组合式专精。尽管单个专家仍具多义性,但专家路径呈现单义性,按语义功能聚类,跨越语言与形式。同一个词元(如“:”)在作为类型标注、引导冒号或时间分隔符时,会遵循不同的语义轨迹。该分解揭示了结构来源:控制子空间中的聚类比完整表示更具单义性。因此,MoE中自然的可解释单元并非专家,而是轨迹。
原文摘要 · Abstract (English)
An LLM's residual stream is both state and instruction: it encodes the current context and determines the next transformation. We introduce a parameter-free decomposition for Mixture-of-Experts models that splits each layer's hidden state into a control signal that causally drives routing and an orthogonal content channel invisible to the router. Across six MoE architectures, we find that models preserve surface-level features (language, token identity, position) in the content channel, while the control signal encodes an abstract function that rotates from layer to layer. Because each routing decision is low-bandwidth, this hand-off forces compositional specialization across layers. While individual experts remain polysemantic, expert paths become monosemantic, clustering tokens by semantic function across languages and surface forms. The same token (e.g., ":") follows distinct trajectories depending on whether it serves as a type annotation, an introductory colon, or a time separator. Our decomposition identifies the source of this structure: clusters in the control subspace are substantially more monosemantic than those in the full representation. As a result, the natural unit of interpretability in MoEs is not the expert but the trajectory.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。