发现掩码扩散模型其实是在学习解码顺序的自回归模型
Masked Diffusion Models are Secretly Learned-Order Autoregressive Models
- 用多变量噪声调度让扩散模型自动学习最优解码顺序
- 理论证明其目标函数等价于加权自回归损失
- 适合研究生成模型机制或想优化采样顺序的学者
掩码扩散模型（MDMs）已成为离散域生成建模中最具前景的方法之一。已知MDMs在训练中以随机顺序解码标记，且该顺序对实际性能有显著影响。这引发一个根本性问题：能否设计一种训练框架来优化解码顺序？本文给出肯定回答，表明当配备多变量噪声调度时，MDMs的连续时间变分目标可识别并优化解码顺序。我们建立了解码顺序与多变量噪声调度之间的直接对应关系，并证明该设定破坏了MDM目标对噪声调度的不变性。此外，我们证明了MDM目标可精确分解为这些顺序上的加权自回归损失，从而确立其为具有可学习顺序的自回归模型。
原文摘要 · Abstract (English)
Masked Diffusion Models (MDMs) have emerged as one of the most promising paradigms for generative modeling over discrete domains. It is known that MDMs effectively train to decode tokens in a random order, and that this ordering has significant performance implications in practice. This observation raises a fundamental question: can we design a training framework that optimizes for a favorable decoding order? We answer this in the affirmative, showing that the continuous-time variational objective of MDMs, when equipped with multivariate noise schedules, can identify and optimize for a decoding order during training. We establish a direct correspondence between decoding order and the multivariate noise schedule and show that this setting breaks invariance of the MDM objective to the noise schedule. Furthermore, we prove that the MDM objective decomposes precisely into a weighted auto-regressive losses over these orders, which establishes them as auto-regressive models with learnable orders.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。