路由机制不影响语言模型性能，简单几何路由也能达到顶尖效果。
Equifinality in Mixture of Experts: Routing Topology Does Not Determine Language Modeling Quality
- 用余弦相似度在低维空间做路由，参数量少80%
- 五种不同路由方式结果相差不到1个困惑度（PPL）
- 适合关注效率与模型轻量化的研究者
稀疏混合专家（MoE）架构不断引入复杂路由机制——学习型路由器、多跳路径、基于令牌的门控。本文探究：路由拓扑是否真正决定语言建模质量？我们构建了基于几何的MoE（ST-MoE），采用余弦相似度路由，在低维空间（$d_{space} = 64$）中对比学习中心点，路由参数减少80%。在WikiText-103上进行62组受控实验（76–84M参数，训练至收敛，50K步，1.64B tokens），发现路由拓扑不影响最终困惑度（PPL）：五种余弦路由变体在1-PPL范围内统计等效（双单侧检验[TOST]，$p < 0.05$，所有10组配对比较；15次运行，3个种子，观察范围33.93–34.72）。该结论扩展至哈希、随机固定和top-1路由（单种子；1.1–2.2 PPL退化），并在OpenWebText上复现（0.03 PPL差距，6次运行，每种子3次）。标准线性路由器参数多5.3倍，达到PPL 32.76，而同参数量的余弦路由缩小67%差距——真实机制优势约1.2%。机制解释为收敛冗余：多跳更新方向高度共线（$ ext{cos}(Δh_0, Δh_1) = 0.805$），实现幅度放大而非组合推理；单个可学习标量即可复现多跳性能。实际收益方面，零样本相对范数截断可节省25% MoE FLOPs，仅增加0.12% PPL。专家级专业化与因果可控性——与拓扑等效性共存——在附录论文中进一步探讨。
原文摘要 · Abstract (English)
Sparse Mixture-of-Experts (MoE) architectures employ increasingly sophisticated routing mechanisms -- learned routers, multi-hop trajectories, token-dependent gating. We ask: does routing topology actually determine language modeling quality? We build a geometric MoE (ST-MoE) using cosine-similarity routing against learned centroids in a low-dimensional space ($d_{space} = 64$), requiring 80% fewer routing parameters than standard linear routers. Through 62 controlled experiments on WikiText-103 at 76--84M parameters trained to convergence (50K steps, 1.64B tokens), we find that routing topology does not determine asymptotic perplexity (PPL): five cosine-routing variants are statistically equivalent within a 1-PPL margin (Two One-Sided Tests [TOST], $p < 0.05$ for all 10 pairwise comparisons; 15 runs across 3 seeds, observed range 33.93--34.72). The finding extends to hash, random-fixed, and top-1 routing (single-seed; graceful 1.1--2.2 PPL degradation) and replicates on OpenWebText (0.03 PPL gap, 6 runs, 3 seeds each). A standard linear router with 5.3$\times$ more routing parameters reaches PPL 32.76, but iso-parameter cosine routing closes 67% of this gap -- the true mechanism advantage is $\sim$1.2%. The mechanistic explanation is convergent redundancy: multi-hop updates are collinear ($\cos(Δh_0, Δh_1) = 0.805$), implementing magnitude amplification rather than compositional reasoning; a single learnable scalar replicates multi-hop performance. As a practical payoff, zero-shot relative-norm halting saves 25% of MoE FLOPs at +0.12% PPL. Expert-level specialization and causal controllability -- which coexist with topology-level equifinality -- are explored in a companion paper.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。