提出无需重训练的推理路由算法,解决专家模型负载不均问题。
From Score Distributions to Balance: Plug-and-Play Mixture-of-Experts Routing
- 根据门控得分分布动态调整专家选择范围
- 在多个数据集上实现更低延迟和更高吞吐量
- 可直接接入现有推理流程,适合部署优化场景
Mixture-of-Experts(MoE)模型通过学习门控函数将每个标记路由到一组专家,从而扩展参数容量。虽然条件路由降低了训练成本,但增加了推理内存负担:专家参数与激活值占用大量内存,限制了每设备的专家数量。令牌路由后,部分专家过载而其他专家利用率低。由于专家映射到GPU,这种不平衡会直接导致系统延迟升高、吞吐量下降和成本增加。我们提出LASER,一种即插即用的推理时路由算法,在保持精度的前提下实现负载均衡。LASER根据门控得分分布自适应调整策略:当得分有明显偏好时,路由至最强专家;当得分较均匀时,扩大候选专家范围,并选择负载最低者。由于LASER仅依赖已训练模型的门控得分,可直接集成至现有MoE推理流程中,无需重新训练或微调。我们在Mixtral-8x7B和DeepSeek-MoE-16b-chat上,针对四个数据集(ARC-Easy、ARC-Challenge、MMLU、GSM8K)进行评估。结果表明,LASER显著改善了负载均衡,带来更低延迟和更高吞吐量,同时精度变化可忽略不计。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) models can scale parameter capacity by routing each token to a subset of experts through a learned gate function. While conditional routing reduces training costs, it shifts the burden on inference memory: expert parameters and activations consume memory, limiting the number of experts per device. As tokens are routed, some experts become overloaded while others are underutilized. Because experts are mapped to GPUs, this imbalance translates directly into degraded system performance in terms of latency, throughput, and cost. We present LASER, a plug-and-play, inference-time routing algorithm that balances load while preserving accuracy. LASER adapts to the shape of the gate's score distribution. When scores provide a clear preference, it routes to the strongest experts; when scores are more uniform, it broadens the set of viable experts and routes to the least-loaded among them. Because LASER relies only on gate scores from a trained model, it integrates directly into existing MoE inference pipelines without retraining or finetuning. We evaluate LASER on Mixtral-8x7B and DeepSeek-MoE-16b-chat across four datasets (ARC-Easy, ARC-Challenge, MMLU, and GSM8K). LASER improves load balancing, translating into lower latency and higher throughput, while keeping the accuracy changes negligible.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。