将1000亿参数的自回归模型转为扩散语言模型，实现高效推理。
LLaDA2.0: Scaling Up Diffusion Language Models to 100B
- 通过三阶段块级扩散训练，渐进式转换自回归模型为扩散模型。
- 100B版本在指令微调后表现优异，支持并行解码提升效率。
- 适合追求高并发、低延迟部署的工业级大模型应用者。
本文提出LLaDA2.0——一组通过系统性转换自回归（AR）模型得到的离散扩散大语言模型（dLLM），总参数规模达1000亿，建立前沿规模部署新范式。不同于从零训练的高昂成本，LLaDA2.0坚持知识继承、渐进适配与效率导向设计原则，采用新型三阶段块级加权采样扩散（WSD）训练方案：块扩散中逐步增大块尺寸（预热）、全序列大规模扩散（稳定）、再回退至紧凑块扩散（衰减）。结合SFT和DPO后训练对齐，获得两个指令微调的MoE变体：LLaDA2.0-mini（16B）与LLaDA2.0-flash（100B），均针对实际部署优化。凭借并行解码优势，两模型在前沿规模下实现卓越性能与效率。两个模型均已开源。
原文摘要 · Abstract (English)
This paper presents LLaDA2.0 -- a tuple of discrete diffusion large language models (dLLM) scaling up to 100B total parameters through systematic conversion from auto-regressive (AR) models -- establishing a new paradigm for frontier-scale deployment. Instead of costly training from scratch, LLaDA2.0 upholds knowledge inheritance, progressive adaption and efficiency-aware design principle, and seamless converts a pre-trained AR model into dLLM with a novel 3-phase block-level WSD based training scheme: progressive increasing block-size in block diffusion (warm-up), large-scale full-sequence diffusion (stable) and reverting back to compact-size block diffusion (decay). Along with post-training alignment with SFT and DPO, we obtain LLaDA2.0-mini (16B) and LLaDA2.0-flash (100B), two instruction-tuned Mixture-of-Experts (MoE) variants optimized for practical deployment. By preserving the advantages of parallel decoding, these models deliver superior performance and efficiency at the frontier scale. Both models were open-sourced.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。