通过笛卡尔积路由提升专家间知识共享，增强大模型性能与鲁棒性。
CartesianMoE: Boosting Knowledge Sharing among Experts via Cartesian Product Routing in Mixture-of-Experts
- 用笛卡尔积方式路由专家，实现专家间更有效的知识融合。
- 在多个任务上降低困惑度，优于现有MoE模型，且对路由误差更鲁棒。
- 适合追求高效知识共享的大规模语言模型研发人员。
大规模语言模型（LLM）因在各类下游任务中表现卓越而备受关注。根据著名的缩放定律，扩大密集模型规模虽能提升能力，但会显著增加计算复杂度。混合专家（MoE）模型通过允许模型规模增长而不显著提高训练或推理成本来解决此问题。然而，MoE模型在专家间知识共享方面面临挑战，导致性能对路由精度敏感。为应对这一问题，先前工作引入共享专家，并以“加法”方式将它们的输出与前K个路由专家的输出结合。本文受集体矩阵分解启发，提出CartesianMoE，以类似“乘法”的方式实现专家间更有效的知识共享。大量实验结果表明，CartesianMoE在降低困惑度和提升下游任务性能方面优于现有MoE模型，且具备更强的专家路由鲁棒性。
原文摘要 · Abstract (English)
Large language models (LLM) have been attracting much attention from the community recently, due to their remarkable performance in all kinds of downstream tasks. According to the well-known scaling law, scaling up a dense LLM enhances its capabilities, but also significantly increases the computational complexity. Mixture-of-Experts (MoE) models address that by allowing the model size to grow without substantially raising training or inference costs. Yet MoE models face challenges regarding knowledge sharing among experts, making their performance somehow sensitive to routing accuracy. To tackle that, previous works introduced shared experts and combined their outputs with those of the top $K$ routed experts in an ``addition'' manner. In this paper, inspired by collective matrix factorization to learn shared knowledge among data, we propose CartesianMoE, which implements more effective knowledge sharing among experts in more like a ``multiplication'' manner. Extensive experimental results indicate that CartesianMoE outperforms previous MoE models for building LLMs, in terms of both perplexity and downstream task performance. And we also find that CartesianMoE achieves better expert routing robustness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。