让大模型专家按输入难易动态调用不同大小的模块，提升效率与性能。
Grove MoE: Towards Efficient and Superior MoE LLMs with Adjugate Experts
- 引入大小不一的专家，根据输入复杂度动态激活
- 330亿参数模型仅激活31.4~32.8亿参数，性能媲美更大模型
- 适合追求高效推理的开源大模型研发者
混合专家（MoE）架构是当前先进大型语言模型的核心。传统MoE使用相同规模的专家，固定激活参数量，无法根据输入复杂度调整，影响计算效率。为此，本文提出Grove MoE，借鉴异构big.LITTLE CPU设计思想，引入大小不同的专家及动态激活机制，实现模型容量扩展的同时控制计算开销。基于此架构，我们通过对Qwen3-30B-A3B-Base模型在中段和后期训练阶段进行升级，构建了两个330亿参数的GroveMoE-Base和GroveMoE-Inst模型。这些模型根据输入词元复杂度动态激活31.4至32.8亿参数，性能达到与同等或更大规模开源模型相当的水平。
原文摘要 · Abstract (English)
The Mixture of Experts (MoE) architecture is a cornerstone of modern state-of-the-art (SOTA) large language models (LLMs). MoE models facilitate scalability by enabling sparse parameter activation. However, traditional MoE architecture uses homogeneous experts of a uniform size, activating a fixed number of parameters irrespective of input complexity and thus limiting computational efficiency. To overcome this limitation, we introduce Grove MoE, a novel architecture incorporating experts of varying sizes, inspired by the heterogeneous big.LITTLE CPU architecture. This architecture features novel adjugate experts with a dynamic activation mechanism, enabling model capacity expansion while maintaining manageable computational overhead. Building on this architecture, we present GroveMoE-Base and GroveMoE-Inst, 33B-parameter LLMs developed by applying an upcycling strategy to the Qwen3-30B-A3B-Base model during mid-training and post-training. GroveMoE models dynamically activate 3.14-3.28B parameters based on token complexity and achieve performance comparable to SOTA open-source models of similar or even larger size.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。