提出一个无量纲参数，可预测专家模型是否健康运行。
E = T*H/(O+B): A Dimensionless Control Parameter for Mixture-of-Experts Ecology
- 用四个超参构造统一控制参数E，判断专家生态状态
- E≥0.5时能彻底避免死专家，无需额外平衡损失
- 适用于视觉与语言任务，且对温度不敏感
我们提出 E = T*H/(O+B)，一个无量纲控制参数，用于预测混合专家（MoE）模型能否维持健康的专家生态或陷入死专家困境。该参数整合了路由温度T、路由熵权重H、真值权重O和平衡权重B四个超参。通过12组受控实验（8个视觉任务，4个语言任务），总计超过11,000个训练轮次，我们验证：仅当E ≥ 0.5时，即可保证零死专家，从而不再需要手工设计的负载均衡辅助损失。该结论在CIFAR-10、CIFAR-100、TinyImageNet-200、WikiText-2和WikiText-103上跨模态验证。六项新发现包括：(1) 死专家可被唤醒，由平衡损失驱动路由重探索；(2) 正交毒性依赖数据集，非普适；(3) 任务复杂度会改变临界E阈值；(4) 模型过拟合与专家生态健康解耦；(5) 三层次MoE自发坍缩为两层次结构；(6) 生态结构在50倍温度范围内保持不变。我们建议E可作为MoE训练的统一诊断工具，类比流体力学中的雷诺数。
原文摘要 · Abstract (English)
We introduce E = T*H/(O+B), a dimensionless control parameter that predicts whether Mixture-of-Experts (MoE) models will develop a healthy expert ecology or collapse into dead experts. E combines four hyperparameters -- routing temperature T, routing entropy weight H, oracle weight O, and balance weight B -- into a single quantity. Through 12 controlled experiments (8 vision, 4 language) totaling over 11,000 training epochs, we establish that E >= 0.5 alone is sufficient to guarantee zero dead experts, removing the necessity for handcrafted load-balancing auxiliary losses. We validate this cross-modally on CIFAR-10, CIFAR-100, TinyImageNet-200, WikiText-2, and WikiText-103. Six additional findings emerge: (1) dead experts can resuscitate -- triggered by balance loss driving router re-exploration; (2) ortho toxicity is dataset-dependent, not universal; (3) task complexity shifts the critical E threshold; (4) model overfitting is decoupled from expert ecological health; (5) three-tier MoE spontaneously collapses into a two-tier functional structure; (6) ecological structure is temperature-invariant across a 50x range. We propose that E serves as a unified diagnostic for MoE training, analogous to the Reynolds number in fluid dynamics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。