arXiv:2608.15787cs.LGcs.AI2026-08

MoE模型路由差异不等于行为影响,需先测暴露度再判断。

Routing Divergence Is Not Evidence of Behavioral Influence in Same-Weight MoE Self-Distillation

  • 分离路由项与内容项,量化路由对输出的影响
  • 路由项仅占输出1.6倍,暴露度为3.2倍,影响有限
  • 适合研究自蒸馏、MoE机制或模型可解释性的读者

相同权重的Mixture-of-Experts(MoE)两次前向传播可能将同一令牌分配给不同专家,这在同权重自蒸馏中构成潜在盲区,即演示条件教师指导仅查询输入的学生。本文研究该不匹配的单步形式,固定权重而非训练轨迹。通过精确的分块分解,分离出固定内容下的路由项与类似密集层的内容项。在七个开源权重检查点和两个领域中,路由项仅占块输出的1.6倍,而其残差流暴露度达3.2倍。在两个验证模型中,持续开启主干网络使暴露度单调变化;共模控制支持质量-一致性机制,而非仅由分母稀释导致。对PubMedQA的预注册补丁显示,完整路由项对输出的影响小于自然上下文效应的一半,且基本可由匹配范数噪声再现,而内容项具有强方向性。规模与合并专家探测表明,窄块级范围非普遍现象,但暴露度在测试边界仍较小。因此,路由器移动本身并非行为影响的证据:应先测量暴露度,仅在决策关键时才采取行为干预。

原文摘要 · Abstract (English)

Two Mixture-of-Experts (MoE) forward passes can share every weight yet route the same token through different experts. This creates a possible blind spot in same-weight self-distillation, where a demonstration-conditioned teacher supervises a query-only student. We study this mismatch in its single-step form, with frozen weights rather than as a proxy for a full training trajectory. An exact blockwise decomposition separates a routing term, which changes gates at fixed content, from a dense-like content term. Across seven open-weight checkpoints and two domains, the routing term spans only $1.6\times$ as a fraction of block output, while its residual-stream exposure spans $3.2\times$. Exposure is ordered by the routed block's share of the residual. Scaling the always-on backbone in two confirmatory models moves exposure monotonically; common-mode controls support a mass-and-coherence mechanism rather than denominator dilution alone. Preregistered PubMedQA patches on three models show that the full routing term moves outputs by less than half the natural context effect and is largely reproduced by matched-norm noise, whereas the content term is strongly direction-specific. Scale and merged-expert probes show that the narrow block-level range is not universal, although exposure remains small at the tested boundaries. Router movement alone is therefore not evidence of behavioral influence: measure exposure first, and use a behavioral intervention when the decision matters.

MoE自蒸馏可解释性路由分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。