Dream 7B用扩散模型实现并行文本生成，效果超越现有模型。
Dream 7B: Diffusion Large Language Models
- 采用扩散模型并行迭代去噪，不依赖自回归逐词生成。
- 在通用、数学、编程任务上均优于现有扩散语言模型。
- 支持任意顺序生成、填空补全，可调质量与速度平衡。
我们提出 Dream 7B，目前最强大的开源扩散型大语言模型。与传统自回归模型逐词生成不同，Dream 7B通过离散扩散建模，在并行迭代去噪中优化序列。该模型在通用、数学和编程任务上持续优于现有扩散语言模型。其表现出更优的规划能力与推理灵活性，包括任意顺序生成、填空补全及可调的质量-速度权衡。这些成果得益于简单但有效的训练方法，如基于自回归大语言模型的初始化和上下文自适应的词级噪声重调度。我们发布了 Dream-Base 与 Dream-Instruct，以促进基于扩散的语言建模研究。
原文摘要 · Abstract (English)
We introduce Dream 7B, the most powerful open diffusion large language model to date. Unlike autoregressive (AR) models that generate tokens sequentially, Dream 7B employs discrete diffusion modeling to refine sequences in parallel through iterative denoising. Our model consistently outperforms existing diffusion language models on general, mathematical, and coding tasks. Dream 7B demonstrates superior planning abilities and inference flexibility, including arbitrary-order generation, infilling capabilities, and tunable quality-speed trade-offs. These results are achieved through simple yet effective training techniques, including AR-based LLM initialization and context-adaptive token-level noise rescheduling. We release both Dream-Base and Dream-Instruct to facilitate further research in diffusion-based language modeling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。