不加参数就能让专家模型避免重复调用，提升效率。
GatePro: Parameter-Free Expert Selection Optimization for Mixture-of-Experts Models
- 通过识别相似专家对并引入局部竞争机制，防止冗余激活。
- 实验显示专家多样性显著提升，能力更互补且无功能重叠。
- 无需额外参数，可随时插入训练过程，适合各类规模模型。
现代大语言模型采用混合专家（MoE）架构实现高效扩展，但面临一个关键挑战：功能相似的专家常被同时选中，导致计算冗余，限制模型有效容量。现有辅助平衡损失方法虽改善了令牌分布，却未能解决专家多样性本质问题。我们提出GatePro，一种全新的无参数方法，直接促进专家选择多样性。GatePro识别最相似的专家对，并引入局部竞争机制，阻止冗余专家共激活，同时保持自然的专家专业化。全面评估表明，GatePro在不同模型规模和基准上均有效。分析显示，其能显著提升专家多样性，使专家发展出更独特且互补的能力，避免功能冗余。该方法可在任意训练阶段热插拔部署，无需额外可学习参数，为提升MoE效能提供实用解决方案。
原文摘要 · Abstract (English)
Modern large language models leverage Mixture-of-Experts (MoE) architectures for efficient scaling, but face a critical challenge: functionally similar experts are often selected simultaneously, creating redundant computation and limiting effective model capacity. Existing auxiliary balance loss methods improve token distribution but fail to address the underlying expert diversity problem. We introduce GatePro, a novel parameter-free method that directly promotes expert selection diversity. GatePro identifies the most similar expert pairs and introduces localized competition mechanisms, preventing redundant expert co-activation while maintaining natural expert specialization. Our comprehensive evaluation demonstrates GatePro's effectiveness across model scales and benchmarks. Analysis demonstrates GatePro's ability to achieve enhanced expert diversity, where experts develop more distinct and complementary capabilities, avoiding functional redundancy. This approach can be deployed hot-swappable during any training phase without additional learnable parameters, offering a practical solution for improving MoE effectiveness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。