通过多头机制融合专家表示空间，提升语言模型性能且保持计算量不变。
MH-MoE: Multi-Head Mixture-of-Experts
- 用多头结构让不同专家协同捕捉多样化特征表示
- 在参数与计算量相同条件下，优于普通MoE和细粒度MoE
- 兼容1比特大模型如BitNet，适合资源受限场景
多头混合专家（MH-MoE）通过多头机制联合关注不同专家所代表的多个表示空间，实现更优性能。本文提出一种新型MH-MoE实现方式，在保持与稀疏混合专家模型相当的计算量（FLOPs）和参数量的前提下，显著提升了语言模型的表现。实验表明，该方法在语言模型上优于标准MoE和细粒度MoE。此外，实验证明MH-MoE可与1比特大语言模型（如BitNet）兼容，具备在低精度部署中的潜力。
原文摘要 · Abstract (English)
Multi-Head Mixture-of-Experts (MH-MoE) demonstrates superior performance by using the multi-head mechanism to collectively attend to information from various representation spaces within different experts. In this paper, we present a novel implementation of MH-MoE that maintains both FLOPs and parameter parity with sparse Mixture of Experts models. Experimental results on language models show that the new implementation yields quality improvements over both vanilla MoE and fine-grained MoE models. Additionally, our experiments demonstrate that MH-MoE is compatible with 1-bit Large Language Models (LLMs) such as BitNet.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。