提出无需路由的专家模型，让每个专家自主决定激活
Routing-Free Mixture-of-Experts
- 专家自主决定激活，无需集中式路由或Softmax等设计
- 通过可调混合机制同时优化专家与令牌负载均衡
- 在多个任务上表现更优，扩展性和鲁棒性更强
标准的混合专家（MoE）模型依赖于集中式路由机制，引入了固定的归纳偏置。我们提出无路由混合专家（Routing-Free MoE），彻底消除外部路由、Softmax、Top-K和负载均衡等硬编码设计，将所有激活功能封装在各个专家内部，并通过连续梯度流直接优化，使每个专家能完全自主决定其激活状态。我们引入统一的自适应负载均衡框架，通过可配置插值同时优化专家平衡与令牌平衡目标，实现灵活可定制的资源分配。大量实验表明，该方法在可扩展性和鲁棒性方面持续优于基线。我们对模型行为进行了深入分析，为未来MoE设计与优化提供洞见。
原文摘要 · Abstract (English)
Standard Mixture-of-Experts (MoE) models rely on centralized routing mechanisms that introduce rigid inductive biases. We propose Routing-Free MoE which eliminates any hard-coded centralized designs including external routers, Softmax, Top-K and load balancing, instead encapsulating all activation functionalities within individual experts and directly optimized through continuous gradient flow, enabling each expert to determine its activation entirely on its own. We introduce a unified adaptive load-balancing framework to simultaneously optimize both expert-balancing and token-balancing objectives through a configurable interpolation, allowing flexible and customizable resource allocation. Extensive experiments show that Routing-Free MoE can consistently outperform baselines with better scalability and robustness. We analyze its behavior in detail and offer insights that may facilitate future MoE design ad optimization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。