改进专家模型路由，让模型提前感知领域切换并精准选专家。
Affinity Is Not Enough: Recovering the Free Energy Principle in Mixture-of-Experts
- 引入记忆机制与预测器，让专家路由能感知领域变化
- 在切换点正确专家概率从0.006提升至0.748，效率提高124倍
- 适合研究神经网络动态路由与认知建模的学者
稀疏混合专家（MoE）路由在领域切换时表现失效，即当前词属于一个分布而下一词属于另一个。在控制实验中（4个专家，5次随机种子），标准亲和力路由在切换点对正确专家的分配概率仅为0.006 ± 0.001。通过三种轻量级门控修改：带有记忆的β机制（累积上下文信息）、精度加权门控（Pi，基于预测误差方差）、以及基于β状态的前瞻预测，该概率提升至0.748 ± 0.002（提升124倍），将实现99%覆盖所需的专家数从不可行降至小常数。消融实验显示β与前瞻存在超加性效应：单独使用无增益（+0.000 ± 0.001），β单独提升+0.295 ± 0.013，两者结合则接近最优（+0.741 ± 0.002），超出相加结果+0.446 ± 0.014。在字符级MoE语言模型中，β路由将切换步的比特每字符（BPC）从6.56 ± 0.01降至4.01 ± 0.15；β + Ant门控可在输入出现前就对正确领域专家赋予0.86 ± 0.02的概率，而标准方法仅0.42 ± 0.12。参考实现约200行代码：https://github.com/russellwmy/affinity-is-not-enough
原文摘要 · Abstract (English)
Sparse MoE routing fails at domain transitions, where the current token belongs to one distribution and the next to another. In a controlled experiment (4 experts, 5 seeds), standard affinity routing assigns only 0.006 +/- 0.001 probability to the correct expert at the transition. Three lightweight gate modifications raise this to 0.748 +/- 0.002 (124x), cutting experts needed for 99% coverage from infeasible to a small constant: temporal memory (beta), a per-expert LIF membrane potential accumulating routing context across tokens; precision-weighted gating (Pi), a per-expert inverse variance of recent prediction error, yielding 31x contrast between reliable and unreliable experts; and anticipatory routing, a next-state predictor conditioned on the beta-accumulated hidden state. The mechanisms draw from Friston's Free Energy Principle and use LIF dynamics from spiking neural networks. An ablation across all 2^3 subsets reveals a super-additive beta x Ant interaction: anticipation alone gives nothing (+0.000 +/- 0.001); beta alone gives modest gain (+0.295 +/- 0.013); combined they close 75% of the oracle gap (+0.741 +/- 0.002, exceeding the sum by +0.446 +/- 0.014). This is structural: a stateless predictor cannot detect approaching transitions because pre-transition tokens are distributionally identical to within-domain tokens. In a character-level MoE LM (5 seeds), beta-routing reduces transition-step BPC from 6.56 +/- 0.01 (Standard) to 4.01 +/- 0.15 (beta-MoE); the beta + Ant gate places 0.86 +/- 0.02 probability on the correct domain expert before that domain appears in input, vs 0.42 +/- 0.12 for Standard MoE. Reference implementations (~200 lines each): https://github.com/russellwmy/affinity-is-not-enough
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。