对比6种路由结构，提升MoE模型的效率与准确性。
Optimizing MoE Routers: Design, Implementation, and Evaluation in Transformer Models
- 设计并实现6种路由架构，优化专家分配机制。
- MLP-Hadamard路由实现结构化稀疏，降低负载不均。
- 适用于需要高效部署的大模型开发者。
Mixture of Experts (MoE) 架构提升大语言模型可扩展性，但其性能依赖于将令牌分配给专用专家的路由模块。不当路由会导致负载不均和准确率下降。本项目在Transformer模型中设计并实现了六种不同的路由架构：Linear、Attention、Multi-Layer Perceptron (MLP)、Hybrid、Hash以及我们提出的新结构MLP-Hadamard。通过BERT和Qwen1.5-MoE模型，评估了这些路由在参数效率、推理延迟、路由熵和专家利用率方面的表现。结果显示：Linear路由速度最快，而MLP和Attention路由更具表达能力；MLP-Hadamard路由展现出独特的结构化稀疏路由能力。我们成功在复杂的量化Qwen1.5-MoE模型中替换并微调了自定义路由。本工作提供了MoE路由设计的比较分析，为大规模模型的高效部署提供了优化思路。
原文摘要 · Abstract (English)
Mixture of Experts (MoE) architectures increase large language model scalability, yet their performance depends on the router module that moves tokens to specialized experts. Bad routing can load imbalance and reduced accuracy. This project designed and implemented different router architectures within Transformer models to fix these limitations. We experimented with six distinct router variants Linear, Attention, Multi-Layer Perceptron (MLP), Hybrid, Hash, and our new MLP-Hadamard. We characterized these routers using BERT and the Qwen1.5-MoE model, looking at parameter efficiency, inference latency, routing entropy, and expert utilization patterns. Our evaluations showed distinct trade-offs: Linear routers offer speed, while MLP and Attention routers provide greater expressiveness. The MLP-Hadamard router shows a unique capability for structured, sparse routing. We successfully replaced and fine-tuned custom routers within the complex, quantized Qwen1.5-MoE model. This work provides a comparative analysis of MoE router designs and offers insights into optimizing their performance for efficient and effective large-scale model deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。