通过智能路由提升MoE大模型性能与推理速度,无需重训练。
Ban&Pick: Enhancing Performance and Efficiency of MoE-LLMs via Smarter Routing
- 后训练阶段动态筛选关键专家并剔除冗余,优化激活策略。
- 在Qwen3-30B-A3B上准确率提升至84.66(AIME2024)和68.18(GPQA-Diamond)。
- 不改架构、不重训练,实现免费性能提升与1.25倍加速。
稀疏混合专家(MoE)已成为高效扩展大语言模型的关键架构。近期细粒度的MoE设计每层引入数百个专家,每个令牌激活多个专家,实现更强的专业化。然而,在预训练阶段,路由器主要优化稳定性与鲁棒性:过早收敛且强制均衡使用,限制了模型在推理阶段的性能与效率潜力。本文发现两个被忽视的问题:(i) 少数高影响力专家因过早且均衡的路由决策而未被充分利用;(ii) 固定每令牌激活专家数量带来显著冗余。为此,我们提出Ban&Pick,一种后训练、即插即用的智能路由策略。Pick识别并强化对性能影响巨大的关键专家组,带来跨领域的显著准确率提升。Ban基于层与令牌敏感度动态剪枝冗余专家,实现更快推理且精度损失极小。在细粒度MoE-LLM(DeepSeek、Qwen3)上,涵盖数学、代码与通用推理基准的实验表明,Ban&Pick在无需重训练或架构改动的前提下,实现了免费性能增益与推理加速。例如,在Qwen3-30B-A3B上,其在AIME2024上准确率从80.67提升至84.66,在GPQA-Diamond上从65.66提升至68.18,同时在vLLM下推理速度提升1.25倍。
原文摘要 · Abstract (English)
Sparse Mixture-of-Experts (MoE) has become a key architecture for scaling large language models (LLMs) efficiently. Recent fine-grained MoE designs introduce hundreds of experts per layer, with multiple experts activated per token, enabling stronger specialization. However, during pre-training, routers are optimized mainly for stability and robustness: they converge prematurely and enforce balanced usage, limiting the full potential of model performance and efficiency at inference. In this work, we uncover two overlooked issues: (i) a few highly influential experts are underutilized due to premature and balanced routing decisions; and (ii) enforcing a fixed number of active experts per token introduces substantial redundancy. Instead of retraining models or redesigning MoE architectures, we introduce Ban&Pick, a post-training, plug-and-play strategy for smarter routing. Pick discovers and reinforces key experts-a small group with outsized impact on performance-leading to notable accuracy gains across domains. Ban further dynamically prunes redundant experts based on layer and token sensitivity, delivering faster inference with minimal accuracy loss. Experiments on fine-grained MoE-LLMs (DeepSeek, Qwen3) across math, code, and general reasoning benchmarks demonstrate that Ban\&Pick delivers free performance gains and inference acceleration without retraining or architectural changes. For instance, on Qwen3-30B-A3B, it improves accuracy from 80.67 to 84.66 on AIME2024 and from 65.66 to 68.18 on GPQA-Diamond, while accelerating inference by 1.25x under the vLLM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。