通过图结构增强专家协作，解决大模型微调中的负载不均问题
GMoE: Empowering LLMs Fine-Tuning via MoE Graph Collaboration
- 设计图路由机制，让专家间动态共享信息
- 在四个数据集上微调效果优于基线，稳定性和效率提升
- 适合需要高效微调大模型的研究者和工程团队
大语言模型的稀疏混合专家（MoE）架构因简单线性路由器策略导致负载不均，进而引发训练不稳定和学习效率低下。为此，我们提出新型基于图的MoE框架GMoE，旨在增强多个专家间的协作。在GMoE中，图路由函数捕捉专家间的协作信号，使所有专家能通过与邻近专家共享信息，动态分配输入数据带来的信息。此外，我们提出了两种协调策略：基于泊松分布的区分策略和基于正态分布的平衡策略，以进一步释放每个专家的潜力，并提高大模型微调时的稳定性。具体地，我们采用参数高效微调技术——低秩适应（LoRA）来实现图式MoE架构。在四个真实世界基准数据集上的大量实验表明，GMoE有效提升了多专家协作能力，验证了其在大模型微调中的优势。实验代码已开源：https://github.com/BAI-LAB/GMoE。
原文摘要 · Abstract (English)
The sparse Mixture-of-Experts (MoE) architecture of large language models (LLMs) confronts an inherent issue of load imbalance arising from the simplistic linear router strategy, which ultimately causes the instability and inefficient learning of LLMs. To address this challenge, we introduce a novel MoE graph-based framework $\textbf{GMoE}$, aimed at enhancing the collaboration among multiple experts. In GMoE, a graph router function is designed to capture the collaboration signals among experts. This enables all experts to dynamically allocate information derived from input data by sharing information with their neighboring experts. Moreover, we put forward two coordination strategies in GMoE: the $\textit{Poisson distribution-based distinction strategy}$ and the $\textit{Normal distribution-based balance strategy}$, to further release the capacity of each expert and increase the model stability in the fine-tuning of LLMs. Specifically, we leverage a parameter-efficient fine-tuning technique, i.e., Low-Rank Adaptation (LoRA), to implement the graph MoE architecture. Extensive experiments on four real-world benchmark datasets demonstrate the effectiveness of GMoE, showing the benefits of facilitating collaborations of multiple experts in LLM fine-tuning. The code of experimental implementation is available at https://github.com/BAI-LAB/GMoE
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。