提出多掩码扩散模型,实现高质量少步生成。
Multi-Mask Diffusion Language Models for Few-Step Generation
- 设计多掩码前向过程,保留掩码结构支持少步生成
- 闭式ELBO训练目标支持从预训练模型持续训练
- 纯离散一致性蒸馏提升生成质量,适合高效训练
掩码扩散模型(MDMs)是一类有前景的语言生成方法,但实现高质量少步生成仍具挑战。在传统MDMs中,所有前向轨迹坍缩至单一全掩码状态,导致终端熵不足,难以支持一致性风格的少步生成。尽管近期基于均匀状态扩散的方法避免了这一退化问题,却更难区分干净词元与噪声,损害建模质量与训练效率。本文提出多掩码扩散模型(MultiMDM),在前向过程中将每个干净词元先推向指定掩码,再逐步混合于掩码集合中,使反向过程具备先预测指定掩码、再精炼为干净词元的草稿能力。我们推导出适用于持续训练的闭式ELBO目标,并提出纯离散状态的一致性蒸馏方案,通过共享Gumbel耦合降低路径熵。预训练与蒸馏实验表明,MultiMDM为原则性少步生成提供了有效基础。
原文摘要 · Abstract (English)
Masked diffusion models (MDMs) are a promising family of language generators, but achieving high-quality few-step generation remains challenging. In MDMs, all forward trajectories collapse to a single fully masked state, leaving no terminal entropy for consistency-style few-step generation. While recent few-step alternatives based on uniform-state diffusion avoid this degeneracy, it becomes harder to distinguish clean tokens from noise than MDMs, which usually harms modeling quality and training efficiency. In this work, we propose a multi-mask diffusion model (MultiMDM) that preserves the masking structure towards few-step generation. In the forward process, each clean token is first pushed towards a designated mask and then gradually mixes over the mask set. As a result, the backward process has a drafting capability by predicting a designated mask before refining to a clean token. We derive a closed-form ELBO training objective for MultiMDM that supports continual training from pretrained MDMs. In addition, we formulate a purely discrete-state consistency distillation scheme, with a shared-Gumbel coupling to reduce pathwise entropy. Experiments on pretraining and distillation show that MultiMDM provides an effective foundation for principled few-step generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。