通过共享专家参数，让大模型在不降性能的前提下大幅减少内存占用。
Tying the Loop -- Tied Expert Layers in Mixture-of-Experts Language Models
- 同一组专家参数跨连续层共享，路由和注意力保持独立。
- 预训练实验显示内存减半，困惑度和下游任务表现几乎不变。
- 适合追求高效训练与部署的LLM研究者与工程师。
Mixture-of-Experts（MoE）架构通过每令牌仅激活少量专家，高效扩展大语言模型，但全量参数（主要由专家参数构成）仍需驻留于训练与推理内存中。为此，我们提出专家绑定（Expert Tying），即在连续Transformer层间共享专家参数，同时保留分层独立的路由与注意力机制。我们在OLMoE、Qwen3及DeepSeek风格的MoE等主流架构上评估该方法。预训练实验表明，专家绑定可使内存占用几乎降低2倍，对困惑度或下游任务质量几乎无影响。通过利用MoE路径中的参数冗余，该方法实现了极优的计算与内存权衡，推动下一代大模型的高效训练与扩展。
原文摘要 · Abstract (English)
Mixture-of-Experts (MoE) architectures efficiently scale Large Language Models (LLMs) by activating only a small fraction of their experts per token, yet the full parameter count - dominated by the expert parameters - must be held in training and inference memory. To address this, we introduce Expert Tying, an architectural modification that shares expert parameters across consecutive transformer layers while preserving independent, layer-wise routing and attention. We evaluate this approach across common, state-of-the-art architectures, including OLMoE, Qwen3, and DeepSeek-style MoEs. Our pretraining experiments demonstrate that tying experts can reduce memory footprint by almost 2x at virtually no degradation in perplexity or downstream quality. By exploiting the parameter redundancy inherent in MoE pathways, our method provides a highly favorable compute-to-memory trade-off, advancing efficient training and scaling of next-generation LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。