提出部分遮蔽机制，让离散扩散模型更高效地生成文本和图像。
Beyond Masked and Unmasked: Discrete Diffusion Models via Partial Masking
- 引入中间状态的局部遮蔽，避免重复处理相同输入。
- 文本任务上困惑度达15.36，优于已有方法；图像生成FID低至3.26。
- 无需自回归结构，适合追求高效生成的场景。
掩码扩散模型（MDM）是强大的离散数据生成模型，通过逐步解码序列中的掩码标记来生成样本。每个标记仅有两种状态：掩码或未掩码。我们观察到，连续采样步骤间标记序列常保持不变，导致模型反复处理相同输入，造成计算冗余。为解决此问题，我们提出部分遮蔽方案（Prime），在传统MDM基础上允许标记处于掩码与未掩码之间的中间状态。该设计使模型基于部分观测信息进行预测，实现更精细的去噪过程。我们推导了变分训练目标，并设计简单架构以支持中间状态输入。实验表明，该方法在多种生成任务中表现优异：在OpenWebText文本数据上，困惑度降至15.36，优于先前的MDM（21.52）、自回归模型（17.54）及其混合模型（17.58），且不依赖自回归结构；在图像数据上，于CIFAR-10取得3.26的FID，ImageNet-32上为6.98，接近领先连续生成模型性能。
原文摘要 · Abstract (English)
Masked diffusion models (MDM) are powerful generative models for discrete data that generate samples by progressively unmasking tokens in a sequence. Each token can take one of two states: masked or unmasked. We observe that token sequences often remain unchanged between consecutive sampling steps; consequently, the model repeatedly processes identical inputs, leading to redundant computation. To address this inefficiency, we propose the Partial masking scheme (Prime), which augments MDM by allowing tokens to take intermediate states interpolated between the masked and unmasked states. This design enables the model to make predictions based on partially observed token information, and facilitates a fine-grained denoising process. We derive a variational training objective and introduce a simple architectural design to accommodate intermediate-state inputs. Our method demonstrates superior performance across a diverse set of generative modeling tasks. On text data, it achieves a perplexity of 15.36 on OpenWebText, outperforming previous MDM (21.52), autoregressive models (17.54), and their hybrid variants (17.58), without relying on an autoregressive formulation. On image data, it attains competitive FID scores of 3.26 on CIFAR-10 and 6.98 on ImageNet-32, comparable to leading continuous generative models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。