arXiv:2608.03457cs.AI2026-08

提出扩散语言模型的专家混合扩展规律,30B模型用更少数据接近Qwen3性能。

LLaDA MoE v2: Scaling Mixture-of-Experts Diffusion Language Models

论文配图:LLaDA MoE v2: Scaling Mixture-of-Experts Diffusion Language Models
图 1 · 摘自论文原文
  • 基于计算分配与优化参数,发现MoE扩散模型有独特缩放规律。
  • 300亿参数模型在23.5万亿词上训练,仅需65%预训练数据即逼近Qwen3水平。
  • 适合追求高效训练、低资源下高性价比的扩散模型研究者。

扩散语言模型(dLLMs)为自回归语言建模提供替代方案,但专家混合(MoE)dLLMs的缩放行为仍不明确。本文系统分析了优化超参数、计算分配与架构对MoE dLLMs的影响,揭示其与自回归模型已知缩放趋势的定量差异:优化方面,最优名义批量大小随计算量增长更快,最优学习率衰减更迅速;模型-数据分配方面,等FLOP分析显示轻微数据侧倾斜:最优令牌预算增长快于激活模型计算量;架构方面,更大规模下固定激活容量时更偏好更大的专家池,中等专家粒度始终有效,且共享专家激活比例在不同规模下保持稳定。基于这些发现,我们从零开始训练了LLaDA MoE v2,一个300亿参数、激活容量300亿的扩散语言模型,在23.5万亿令牌上进行训练。相比Qwen3,其预训练数据量约为后者的65%,但在多项知识、推理与编程基准测试中表现接近。仅经监督微调后,其在八个推理与编程任务中的七个上超越SDAR Chat,部分任务接近Qwen3。结果确立了MoE dLLMs的实用缩放法则与设计原则。

原文摘要 · Abstract (English)

Diffusion language models (dLLMs) offer an alternative to autoregressive (AR) language modeling, yet the scaling behavior of Mixture-of-Experts (MoE) dLLMs remains poorly understood. We systematically characterize how optimization hyperparameters, compute allocation, and architecture scale for MoE dLLMs, identifying quantitative differences from scaling trends previously reported for AR models. Specifically, for optimization, the optimal nominal batch size grows faster, while the optimal learning rate decays more rapidly with compute. For model--data allocation, IsoFLOP analysis reveals a slight data-side tilt: the optimal token budget grows faster than activated model-side computation. For MoE architecture, larger scales increasingly favor larger expert pools at fixed activated capacity, while moderate expert granularity remains consistently effective and the preferred fraction of activated capacity assigned to shared experts remains stable across scales. Guided by these findings, we train LLaDA MoE v2, a 30B-A3B dLLM, from scratch on 23.5T tokens. With approximately 65\% as many pretraining tokens as Qwen3, LLaDA MoE v2 approaches Qwen3 on several knowledge, reasoning, and coding benchmarks. After supervised fine-tuning alone, it outperforms SDAR Chat on seven of eight reasoning and coding benchmarks and remains close to Qwen3 on several tasks. These results establish practical scaling laws and design principles for MoE dLLMs.

扩散模型专家混合语言模型缩放定律

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。