提出新路由方法，让专家模型各尽其用，计算效率大提升
Latent Prototype Routing: Achieving Near-Perfect Load Balancing in Mixture-of-Experts
- 基于聚类思想设计新型路由机制，动态分配专家任务
- 专家负载不均衡程度降低至0.035，最低与最高负载比达0.70
- 适配多个主流MoE模型，提升算力利用率且不影响性能
混合专家（MoE）架构已成为高效扩展大型语言模型的关键策略。然而，当前MoE系统存在严重负载不均问题，训练和推理过程中仅少数专家被持续激活，导致模型容量和计算资源大量闲置。本文从聚类视角重新审视专家路由，提出一种名为潜在原型路由（Latent Prototype Routing, LPR）的新框架，该方法在保持下游性能的前提下，显著提升专家利用均衡性。在DeepSeek-V3、Qwen3-MoE和Mixtral等多个开源MoE模型上的实验表明，LPR将专家负载的吉尼系数从平均0.70降至0.035，最小与最大专家负载比从1e-6提升至0.70，实现了近乎完美的负载均衡。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) architectures have emerged as a key strategy for scaling large language models (LLMs) efficiently. However, current MoE systems suffer from severe load imbalance, where only a small subset of experts is consistently activated during training and inference, leading to significant underutilization of model capacity and computational resources. In this work, we revisit expert routing through a clustering perspective and propose Latent Prototype Routing (LPR), a novel routing framework that generalizes existing approaches while promoting balanced expert utilization without compromising downstream performance. Extensive experiments across multiple open-source MoE models -- including DeepSeek-V3, Qwen3-MoE, and Mixtral -- demonstrate that LPR reduces the Gini coefficient of expert load from 0.70 to 0.035 on average, improves the min-max expert load ratio from 1e-6 to 0.70, achieving near-perfect load balancing.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。