提出统一框架,让专家模型更智能地分配计算资源。
Rethinking Sparse Mixture of Experts from a Unified Perspective
- 用线性规划统一建模,动态优化令牌与专家匹配。
- 跨数据集、任务和训练方式显著提升性能,最高增益达12.3%。
- 支持灵活预算配置,推理成本降低且不牺牲效果,适合实际部署。
稀疏混合专家(SMoE)模型在保持计算开销恒定的同时扩展了模型容量。现有SMoE方法分为两类:令牌选择(每个令牌分配固定数量专家)和专家选择(每个专家接收固定数量令牌)。然而,固定的预算导致两类方法常选无关的令牌-专家对或遗漏关键分配,从而降低整体性能。为此,本文从线性规划视角重新思考SMoE,提出统一框架。我们引入统一稀疏混合专家(USMoE),包含统一机制与统一评分函数,以克服上述限制。理论分析与实证结果均表明其有效性。在多种数据设置(干净与损坏数据)、多个领域(文本与视觉任务)及不同学习方式(免训练与基于训练)下的广泛评估显示,USMoE不仅显著优于现有SMoE方法,还支持更灵活的专家选择预算,在不损失性能的前提下降低推理成本。代码已开源:https://github.com/giangdip2410/USMoE。
原文摘要 · Abstract (English)
Sparse Mixture of Experts (SMoE) models scale the capacity of models while maintaining constant computational overhead. SMoE methods fall into two categories: Token Choice, which routes each token to a fixed number of experts, and Expert Choice, which assigns a fixed number of tokens to each expert. However, the use of fixed budgets for tokens or experts causes both approaches to select irrelevant token-expert pairs or overlook critical assignments, which degrades overall performance. To fill that gap, we rethink SMoE from a unified perspective through the lens of linear programming, which provides a general formulation for SMoE models. Furthermore, we introduce Unified Sparse Mixture of Experts (USMoE), a novel framework comprising a unified mechanism and a unified score to overcome these limitations. We provide both theoretical justification and empirical evidence demonstrating USMoE's effectiveness. Extensive evaluations across diverse data settings (clean and corrupted), multiple domains (including texts and vision tasks), and different learning approaches (training-free and training-based) show that USMoE not only delivers significant performance improvements over existing SMoE methods, but also enables more flexible expert selection budgets, reducing inference costs without compromising model performance. Our implementation is publicly available at https://github.com/giangdip2410/USMoE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。