超大规模开源专家混合模型，专为企事业单位场景优化。
Yuan3.0 Ultra: A Trillion-Parameter Enterprise-Oriented MoE LLM
- 提出层自适应专家剪枝算法，训练时动态裁剪低效专家并重分配计算资源。
- 预训练效率提升49%，参数总量减少33.3%，仍保持多领域顶尖性能。
- 在文档处理、问答、摘要等企业任务中表现领先，适合工业级应用。
我们推出Yuan3.0 Ultra，一个开源的专家混合（MoE）大语言模型，激活参数达688亿，总参数达10100亿，专为提升企业场景任务表现而设计，同时保持通用任务竞争力。提出面向MoE大模型预训练阶段的层自适应专家剪枝（LAEP）算法。与以往主要在后训练阶段进行剪枝的方法不同，该算法通过统计令牌分布，动态裁剪低效专家并重新组织跨设备专家分布，显著提升训练效率。实验表明，从原始15150亿参数模型开始训练，LAEP实现49%的预训练效率提升和33.3%的总参数量减少，同时维持模型卓越的多领域性能。在文档匹配、ChatRAG、SummEval和MMTab等企业基准测试中，Yuan3.0 Ultra达到领先准确率。模型与代码已公开于https://github.com/Yuan-lab-LLM/Yuan3.0-Ultra。
原文摘要 · Abstract (English)
We introduce Yuan3.0 Ultra, an open-source Mixture-of-Experts (MoE) large language model featuring 68.8B activated parameters and 1010B total parameters, specially designed to enhance performance on enterprise scenarios tasks while maintaining competitive capabilities on general purpose tasks. We propose Layer-Adaptive Expert Pruning (LAEP) algorithm designed for the pre-training stage of MoE LLMs. In contrast to previous expert pruning approaches that operate primarily in the post-training phase, the proposed algorithm enhances training efficiency by selectively pruning underutilized experts and reorganizing experts across computing devices according to token distribution statistics. Comprehensive experiments demonstrate that LAEP effectively reduces model size and substantially improves pre-training efficiency. When pre-training Yuan3.0 Ultra from scratch original with 1515B parameters, this algorithm delivers a 49\% boost in pre-training efficiency and a 33.3\% reduction in total parameters, while preserving the model's outstanding multi-domain performance. On enterprise scenario benchmarks including Docmatix, ChatRAG, SummEval and MMTab, Yuan3.0 Ultra achieves leading accuracy. The model and codes are publicly available at https://github.com/Yuan-lab-LLM/Yuan3.0-Ultra.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。