用扩散模型实现语言生成，性能媲美自回归大模型。
Large Language Diffusion Models
- 采用前向掩码+反向生成的扩散框架，用Transformer预测被掩码词。
- 80亿参数版本在上下文学习上接近LLaMA3，指令遵循能力出色。
- 解决逆向难题，诗歌补全任务超越GPT-4o，挑战自回归范式主导地位。
大型语言模型（LLMs）的能力通常被认为依赖于自回归建模（ARM）。我们通过提出从零训练的扩散模型LLaDA，挑战这一观点。LLaDA采用前向数据掩码与反向生成过程，由Transformer参数化以预测被掩码的词元，通过优化似然下界提供合理的概率推断方法。在通用任务、数学、代码等多个基准测试中，LLaDA展现出强可扩展性，性能与自建的自回归基线相当。值得注意的是，LLaDA 8B在上下文学习方面与强大模型如LLaMA3 8B相当，经过监督微调（SFT）后，在多轮对话等案例研究中表现出色。此外，其克服了逆向难题，在逆向诗歌补全任务中超越GPT-4o。结果表明，扩散模型在大规模语言建模中具有巨大潜力，且前述核心能力并不必然依赖于自回归机制。项目页面与代码：https://ml-gsai.github.io/LLaDA-demo/
原文摘要 · Abstract (English)
The capabilities of large language models (LLMs) are widely regarded as relying on autoregressive models (ARMs). We challenge this notion by introducing LLaDA, a diffusion model trained from scratch under the pre-training and supervised fine-tuning (SFT) paradigm. LLaDA employs a forward data masking process and a reverse generation process, parameterized by a Transformer to predict masked tokens. It provides a principled generative approach for probabilistic inference by optimizing a likelihood lower bound. Across extensive benchmarks on general tasks, math, code, and so on, LLaDA demonstrates strong scalability and performs comparably to our self-constructed ARM baselines. Remarkably, LLaDA 8B is competitive with strong LLMs like LLaMA3 8B in in-context learning and, after SFT, exhibits impressive instruction-following abilities in case studies such as multi-turn dialogue. Moreover, LLaDA addresses the reversal curse, surpassing GPT-4o in a reversal poem completion task. Our findings show the promise of diffusion models for language modeling at scale and challenge the common assumption that core LLM capabilities discussed above inherently depend on ARMs. Project page and codes: https://ml-gsai.github.io/LLaDA-demo/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。