8B参数的双向扩散语言模型，训练效果超越自回归模型。
Improved Large Language Diffusion Models
- 用全双向注意力训练，全程保持掩码扩散目标。
- 预训练12万亿词，微调250亿词指令数据，多任务表现显著提升。
- 适合追求高效非自回归生成与强推理能力的研究者。
现代大语言模型主要采用自回归因子化和因果注意力进行训练。本文提出iLLaDA，一个从零开始训练的80亿参数掩码扩散语言模型，全程使用全双向注意力。iLLaDA在预训练和监督微调中均保持掩码扩散目标，预训练规模达12万亿词，微调使用250亿词的指令语料库，共训练12轮。为提高效率，引入变长生成策略，并采用置信度评分法进行多选题评估。相比原版LLaDA，iLLaDA在通用、数学和代码基准上均有显著提升：iLLaDA-Base在BBH上提升21.6分，在ARC-Challenge上提升14.9分；iLLaDA-Instruct在MATH上提升14.5分，在HumanEval上提升16.5分。尽管采用非自回归训练，其性能仍可媲美Qwen2.5 7B。结果表明，从零开始的全双向扩散训练是构建高性能语言模型的有效路径。模型权重与代码见：https://github.com/ML-GSAI/LLaDA。
原文摘要 · Abstract (English)
Modern large language models are predominantly trained with autoregressive factorization and causal attention. We present \emph{iLLaDA}, an 8B masked diffusion language model trained from scratch with fully bidirectional attention. iLLaDA keeps the masked diffusion objective throughout pre-training and supervised fine-tuning (SFT), scaling pre-training to 12T tokens and fine-tuning on a 25B-token instruction corpus for 12 epochs. We further use variable-length generation for efficiency and introduce confidence-based scoring for multiple-choice evaluation. Compared with LLaDA, iLLaDA improves broadly across general, mathematical, and code benchmarks; for example, iLLaDA-Base improves by 21.6 points on BBH and 14.9 points on ARC-Challenge, while iLLaDA-Instruct improves by 14.5 points on MATH and 16.5 points on HumanEval. Despite its non-autoregressive training, iLLaDA also remains competitive with Qwen2.5 7B on several benchmarks. These results show that fully bidirectional diffusion training from scratch is a competitive path toward strong language models. Model weights and codes: https://github.com/ML-GSAI/LLaDA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。