揭示了MoE路由中负载失衡的最小动力学机制。
A Minimal Bifurcation Model of Load Imbalance in a Softmax Mixture-of-Experts Router
- 基于强化学习规则构建简化动力学模型，模拟专家选择与评分衰减。
- 发现反馈过强时系统出现对称破缺，导致负载失衡的突变。
- 适用于理解小规模MoE模型中的负载不均问题，适合模型设计者参考。
我们提出一个针对双专家混合专家（MoE）层的自适应softmax路由的最小动力学模型。该模型源于离散强化规则的平均场极限：被选中的专家得分微增，所有得分则经历正则化衰减。在对称情况下，极限系统表现出超临界Pitchfork分岔：弱反馈下存在唯一稳定平衡态；超过临界反馈强度后，出现两个稳定的非对称态。当引入外部不对称性时，分岔结构演变为一对折叠分岔，在控制参数平面上形成尖点奇点。我们推导出分岔集的精确参数方程及尖点灾变的局部标准型。数值实验将这一理论框架与实际专家负载关联，涵盖小型可训练MoE模型、硬Top-1 PyTorch路由以及手写数字分类实验。结果揭示了一种可控的低维机制，解释了自适应MoE路由中负载失衡的突发性转变。
原文摘要 · Abstract (English)
We propose a minimal dynamical model of adaptive softmax routing for a two-expert Mixture-of-Experts (MoE) layer. The model is obtained as a mean-field limit of a discrete reinforcement rule: the selected expert receives a small score increment, while all scores undergo regularizing decay. In the symmetric case the limiting system has a supercritical pitchfork bifurcation: for weak feedback there is a unique stable balanced state, whereas above a critical feedback strength two stable asymmetric states appear. When an external asymmetry is added, the pitchfork unfolds into a pair of fold bifurcations forming a cusp in the control-parameter plane. We derive exact parametric equations for the bifurcation set and the local normal form of the cusp catastrophe. Numerical experiments connect this picture to empirical expert load, a small trainable MoE model, hard top-1 PyTorch routing, and a small classification experiment on digits. The results provide a controlled low-dimensional mechanism for abrupt transitions to load imbalance in adaptive MoE routers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。