通过自反思机制增强专家网络的认知深度，提升语言模型推理能力。
GRAPHMOE: Amplifying Cognitive Depth of Mixture-of-Experts Network via Introducing Self-Rethinking Mechanism
- 引入伪图结构的循环路由策略，模拟多轮思考过程
- 在多个基准数据集上超越现有基于LoRA的模型，达当前最优水平
- 适合关注大模型推理能力提升与专家网络优化的研究者
传统混合专家（MoE）网络通过多个小型专家模型替代单一大型网络获得优势，但这些专家通常独立运作，其相互连接是否能提升性能仍待探索。为此，本文提出GRAPHMOE，一种基于伪图MoE网络构建自反思机制的新方法，旨在增强语言模型的认知深度。GRAPHMOE采用循环路由策略模拟迭代思考过程，促进专家节点间的信息流动。通过低秩适配（LoRA）实现该架构，并在多个基准数据集上进行大量实验。结果表明，GRAPHMOE优于其他基于LoRA的模型，在多项任务中达到当前最优（SOTA）性能。此外，本研究提出的新型循环路由策略为提升语言模型推理能力提供了新思路。
原文摘要 · Abstract (English)
Traditional Mixture-of-Experts (MoE) networks benefit from utilizing multiple smaller expert models as opposed to a single large network. However, these experts typically operate independently, leaving a question open about whether interconnecting these models could enhance the performance of MoE networks. In response, we introduce GRAPHMOE, a novel method aimed at augmenting the cognitive depth of language models via a self-rethinking mechanism constructed on Pseudo GraphMoE networks. GRAPHMOE employs a recurrent routing strategy to simulate iterative thinking steps, thereby facilitating the flow of information among expert nodes. We implement the GRAPHMOE architecture using Low-Rank Adaptation techniques (LoRA) and conduct extensive experiments on various benchmark datasets. The experimental results reveal that GRAPHMOE outperforms other LoRA based models, achieving state-of-the-art (SOTA) performance. Additionally, this study explores a novel recurrent routing strategy that may inspire further advancements in enhancing the reasoning capabilities of language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。