让专家模型无限扩展，用连续采样提升效率与性能。
$\infty$-MoE: Generalizing Mixture of Experts to Infinite Experts
- 用连续值采样代替离散选择，实现无限专家的可训练架构。
- 129万活跃参数达中等模型350万参数的性能，推理时可调精度与速度。
- 适合需要灵活部署、追求高性价比的生成式模型应用。
Mixture of Experts（MoE）通过每令牌选择少数前馈网络，在计算成本与性能间取得平衡。传统MoE将每个专家视为独立个体，并在离散空间组合，导致专家数量增加时难以有效训练。为此，我们提出∞-MoE，基于每令牌连续采样的方式，从大前馈网络中选取部分参数。通过将专家置于连续空间，该方法可在保持计算效率的同时支持无限专家。实验表明，基于GPT-2 Small的∞-MoE模型，拥有129万活跃参数和186万总参数，性能可媲美拥有350万参数的密集型GPT-2 Medium。推理时调节采样专家数，可在准确率与速度间灵活权衡，相较传统MoE最高提升2.5%准确率。
原文摘要 · Abstract (English)
The Mixture of Experts (MoE) selects a few feed-forward networks (FFNs) per token, achieving an effective trade-off between computational cost and performance. In conventional MoE, each expert is treated as entirely independent, and experts are combined in a discrete space. As a result, when the number of experts increases, it becomes difficult to train each expert effectively. To stabilize training while increasing the number of experts, we propose $\infty$-MoE that selects a portion of the parameters of large FFNs based on continuous values sampled for each token. By considering experts in a continuous space, this approach allows for an infinite number of experts while maintaining computational efficiency. Experiments show that a GPT-2 Small-based $\infty$-MoE model, with 129M active and 186M total parameters, achieves comparable performance to a dense GPT-2 Medium with 350M parameters. Adjusting the number of sampled experts at inference time allows for a flexible trade-off between accuracy and speed, with an improvement of up to 2.5\% in accuracy over conventional MoE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。