用自回归模型改造扩散语言模型，实现高效生成与推理。
Scaling Diffusion Language Models via Adaptation from Autoregressive Models
- 从自回归模型持续预训练，构建扩散语言模型
- 127M到7B参数模型均成功转换，训练量<200B tokens
- 生成流畅、支持上下文学习，适合需要稳定输出的场景
扩散语言模型（DLMs）作为文本生成的新范式，有望克服自回归（AR）模型的局限。然而，当前DLMs规模远小于其AR对应模型，且缺乏在语言建模基准上的公平比较。此外，大规模训练扩散模型仍具挑战。鉴于开源AR语言模型的广泛存在，本文提出通过适配现有AR模型来构建文本扩散模型。我们揭示了AR与扩散建模目标之间的联系，并引入一种简单的持续预训练方法。在语言建模、推理和常识推理基准上系统评估表明，可将127M至7B参数的GPT2和LLaMA模型转化为DiffuGPT和DiffuLLaMA，仅需不到200B tokens训练。实验结果显示，这些模型优于早期DLMs，且与对应AR模型具有竞争力。我们发布了涵盖127M-355M-7B参数的DLMs系列，具备流畅生成、上下文学习、无需重排提示即可补全中间内容及遵循指令的能力，详见https://github.com/HKUNLP/DiffuLLaMA。
原文摘要 · Abstract (English)
Diffusion Language Models (DLMs) have emerged as a promising new paradigm for text generative modeling, potentially addressing limitations of autoregressive (AR) models. However, current DLMs have been studied at a smaller scale compared to their AR counterparts and lack fair comparison on language modeling benchmarks. Additionally, training diffusion models from scratch at scale remains challenging. Given the prevalence of open-source AR language models, we propose adapting these models to build text diffusion models. We demonstrate connections between AR and diffusion modeling objectives and introduce a simple continual pre-training approach for training diffusion models. Through systematic evaluation on language modeling, reasoning, and commonsense benchmarks, we show that we can convert AR models ranging from 127M to 7B parameters (GPT2 and LLaMA) into diffusion models DiffuGPT and DiffuLLaMA, using less than 200B tokens for training. Our experimental results reveal that these models outperform earlier DLMs and are competitive with their AR counterparts. We release a suite of DLMs (127M-355M-7B) capable of generating fluent text, performing in-context learning, filling in the middle without prompt re-ordering, and following instructions https://github.com/HKUNLP/DiffuLLaMA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。