将稀疏MoE架构融入扩散语言模型，推理时仅激活14亿参数却保持高性能。
LLaDA-MoE: A Sparse MoE Diffusion Language Model
- 采用稀疏MoE架构，训练时用20万亿词，推理仅激活14亿参数。
- 在多个评测中超越更大模型的扩散语言模型，性能达当前最佳。
- 适合追求高效推理与低资源部署的开发者，尤其关注生成质量者。
我们提出 LLaDA-MoE，一个基于混合专家（MoE）架构的大型扩散语言模型，从头训练于约20万亿令牌数据。该模型在保持70亿参数容量的同时，推理时仅激活14亿参数，显著降低计算开销。实证评估显示，LLaDA-MoE在多个基准测试中表现优于此前的扩散语言模型（如 LLaDA、LLaDA 1.5、Dream），达到当前最佳水平。经指令微调的 LLaDA-MoE-7B-A1B-Instruct 模型在知识理解、代码生成、数学推理、智能体任务和对齐能力方面，表现接近 Qwen2.5-3B-Instruct，尽管其活跃参数更少。结果表明，在掩码扩散语言模型的训练目标中引入稀疏MoE架构仍能发挥其高效推理优势，为扩散语言模型的进一步探索提供了广阔空间。模型已开源至 Huggingface。
原文摘要 · Abstract (English)
We introduce LLaDA-MoE, a large language diffusion model with the Mixture-of-Experts (MoE) architecture, trained from scratch on approximately 20T tokens. LLaDA-MoE achieves competitive performance with significantly reduced computational overhead by maintaining a 7B-parameter capacity while activating only 1.4B parameters during inference. Our empirical evaluation reveals that LLaDA-MoE achieves state-of-the-art performance among diffusion language models with larger parameters, surpassing previous diffusion language models LLaDA, LLaDA 1.5, and Dream across multiple benchmarks. The instruct-tuned model LLaDA-MoE-7B-A1B-Instruct demonstrates capabilities comparable to Qwen2.5-3B-Instruct in knowledge understanding, code generation, mathematical reasoning, agent and alignment tasks, despite using fewer active parameters. Our results show that integrating a sparse MoE architecture into the training objective of masked diffusion language models still brings out MoE's strengths under efficient inference with few active parameters, and opens ample room for further exploration of diffusion language models. LLaDA-MoE models are available at Huggingface.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。