用离散向量量化替代传统路由，提升专家模型的稳定性和性能。
On the Role of Discrete Representation in Sparse Mixture of Experts
- 通过向量量化生成输入的离散表示，间接分配专家
- 在多任务中实现28%的鲁棒性提升，微调表现优异
- 适合追求稳定高效的大型模型专家系统设计者
稀疏专家混合（SMoE）是提升模型容量而不增加计算成本的有效方法。其关键组件路由器负责将输入分配给相关专家，但常引发路由不一致和表征坍缩问题。不同于以往修复路由器的做法，本文提出一种新思路：通过输入的离散表示间接指派专家，该表示通过向量量化学习获得，形成新的架构——向量量化专家混合（VQMoE）。我们提供了理论支持与实证证据，证明VQMoE能有效克服传统路由器的挑战。在大规模语言模型和视觉任务的预训练与微调中广泛评估显示，相比其他SMoE路由方法，VQMoE在鲁棒性上提升28%，同时保持微调任务中的强性能。
原文摘要 · Abstract (English)
Sparse mixture of experts (SMoE) is an effective solution for scaling up model capacity without increasing the computational costs. A crucial component of SMoE is the router, responsible for directing the input to relevant experts; however, it also presents a major weakness, leading to routing inconsistencies and representation collapse issues. Instead of fixing the router like previous works, we propose an alternative that assigns experts to input via indirection, which employs the discrete representation of input that points to the expert. The discrete representations are learnt via vector quantization, resulting in a new architecture dubbed Vector-Quantized Mixture of Experts (VQMoE). We provide theoretical support and empirical evidence demonstrating the VQMoE's ability to overcome the challenges present in traditional routers. Through extensive evaluations on both large language models and vision tasks for pre-training and fine-tuning, we show that VQMoE achieves a 28% improvement in robustness compared to other SMoE routing methods, while maintaining strong performance in fine-tuning tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。