S2MoE通过随机学习提升专家模型鲁棒性，减少计算开销28%。
S2MoE: Robust Sparse Mixture of Experts via Stochastic Learning
- 引入不确定性下的随机学习机制，兼顾确定与非确定输入
- 在多个任务上性能媲美现有方法，推理计算量降低28%
- 适合追求高效推理的大模型应用，如资源受限场景
稀疏专家混合模型（SMoE）通过将输入标记路由到少数专家，实现大语言模型的高效训练。然而，训练过程面临表征坍塌问题。现有方法虽聚焦于改进路由器，但仍存在两大局限：（1）专家嵌入维度远小于模型整体维度，加剧表征坍塌；（2）每个输入路由至前K个专家会导致其学习相似特征。本文提出一种新方法——基于随机学习的鲁棒稀疏专家混合模型（S2MoE），该模型通过在不确定性下学习，从确定性和非确定性输入中同时提取信息。大量实验表明，S2MoE在多种任务上表现接近其他路由方法，同时推理计算成本降低28%。
原文摘要 · Abstract (English)
Sparse Mixture of Experts (SMoE) enables efficient training of large language models by routing input tokens to a select number of experts. However, training SMoE remains challenging due to the issue of representation collapse. Recent studies have focused on improving the router to mitigate this problem, but existing approaches face two key limitations: (1) expert embeddings are significantly smaller than the model's dimension, contributing to representation collapse, and (2) routing each input to the Top-K experts can cause them to learn overly similar features. In this work, we propose a novel approach called Robust Sparse Mixture of Experts via Stochastic Learning (S2MoE), which is a mixture of experts designed to learn from both deterministic and non-deterministic inputs via Learning under Uncertainty. Extensive experiments across various tasks demonstrate that S2MoE achieves performance comparable to other routing methods while reducing computational inference costs by 28%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。