通过低维隐空间分解专家模型，显著降低大语言模型参数开销。
MoLAE: Mixture of Latent Experts for Parameter-Efficient Language Models
- 将专家模块分解为共享投影与专属变换，减少参数量
- 在相同性能下，内存和计算开销大幅下降
- 适合资源受限场景下的大模型高效部署
Mixture of Experts（MoE）已成为高效扩展大语言模型的关键架构范式，通过为每个输入词元仅激活部分参数来提升效率。然而，标准MoE架构面临高内存占用和分布式训练中通信开销大的问题。本文提出一种新型参数化方法——隐空间专家混合（MoLAE），通过将专家操作重构为共享投影至低维隐空间，再进行专家特定变换，实现参数分解。该方法显著降低参数量与计算需求，尤其适用于隐藏层维度远大于MoE中间维度的现有大模型。我们构建了严格的数学框架，用于将预训练的MoE模型转化为MoLAE结构，明确了最优分解条件，并设计了一套两步系统性转换算法。理论分析表明，MoLAE在多个维度上均大幅提升效率且保持模型能力。实验结果验证，MoLAE在性能接近标准MoE的同时，资源消耗显著降低。
原文摘要 · Abstract (English)
Mixture of Experts (MoE) has become a key architectural paradigm for efficiently scaling Large Language Models (LLMs) by selectively activating a subset of parameters for each input token. However, standard MoE architectures face significant challenges, including high memory consumption and communication overhead during distributed training. In this paper, we introduce Mixture of Latent Experts (MoLAE), a novel parameterization that addresses these limitations by reformulating expert operations through a shared projection into a lower-dimensional latent space, followed by expert-specific transformations. This factorized approach substantially reduces parameter count and computational requirements, particularly in existing LLMs where hidden dimensions significantly exceed MoE intermediate dimensions. We provide a rigorous mathematical framework for transforming pre-trained MoE models into MoLAE architecture, characterizing conditions for optimal factorization, and developing a systematic two-step algorithm for this conversion. Our comprehensive theoretical analysis demonstrates that MoLAE significantly improves efficiency across multiple dimensions while preserving model capabilities. Experimental results confirm that MoLAE achieves comparable performance to standard MoE with substantially reduced resource requirements.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。