统一多种生成顺序的扩散模型，提升文本生成质量。
Unifying Masked Diffusion Models with Various Generation Orders and Beyond
- 提出OeMDM框架，统一解释各类扩散与自回归生成机制。
- LoMDM联合学习生成顺序与模型，实现上下文相关顺序生成。
- 在多个语言模型基准上优于现有离散扩散模型。
掩码扩散模型（MDMs）是语言生成中替代自回归模型（ARMs）的潜在方案，但生成质量高度依赖生成顺序。以往方法要么硬编码顺序（如块级从左到右），要么为预训练的MDM学习顺序策略，导致额外开销且两阶段优化易得次优解。为此，我们提出有序表达掩码扩散模型（OeMDM），适用于多种生成顺序的广义扩散生成过程，可统一解释MDM、ARM与块扩散。进一步基于OeMDM，我们引入可学习顺序掩码扩散模型（LoMDM），通过单一目标从零开始联合学习生成顺序与扩散主干，实现上下文相关的生成顺序。实验证明，LoMDM在多个语言建模基准上优于多种离散扩散模型。
原文摘要 · Abstract (English)
Masked diffusion models (MDMs) are a potential alternative to autoregressive models (ARMs) for language generation, but generation quality depends critically on the generation order. Prior work either hard-codes an ordering (e.g., blockwise left-to-right) or learns an ordering policy for a pretrained MDM, which incurs extra cost and can yield suboptimal solutions due to the two-stage optimization. Motivated by this, we propose order-expressive masked diffusion model (OeMDM) for a broad class of diffusion generative processes with various generation orders, enabling the interpretation of MDM, ARM, and block diffusion in a single framework. Furthermore, building on OeMDM, we introduce learnable-order masked diffusion model (LoMDM), which jointly learns the generation ordering and diffusion backbone through a single objective from scratch, enabling the diffusion model to generate text in context-dependent ordering. Empirically, we confirm that LoMDM outperforms various discrete diffusion models across multiple language modeling benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。