直接在离散空间做扩散模型,用概率预测生成文本图像。
In-Situ Tweedie Discrete Diffusion Models
- 直接在独热向量上加噪声并迭代去噪,不依赖连续嵌入或掩码。
- 在图像分类和文本生成任务上表现优秀,优于现有离散扩散方法。
- 统一分类与生成,适合需要原生离散建模的场景。
尽管扩散模型在生成连续数据(如图像)方面表现出色,但将其应用于离散任务时,通常依赖间接方法,如在连续嵌入空间中操作或使用标记掩码,这些方法偏离了由Tweedie公式保证的真实离散数据分布。本文提出原位Tweedie离散扩散(TDD),一种在离散独热空间中直接执行、由Tweedie公式保障的扩散框架。与以往通过连续嵌入或掩码进行扩散的方法不同,TDD直接对独热向量添加高斯噪声,并通过时间步条件化的交叉熵目标进行迭代去噪,而非均方误差重建。每一步去噪中,模型预测类别概率,取argmax得到离散预测,转为独热向量后输入下一轮,噪声逐步降低。该过程自然统一了判别分类与生成建模。实验表明,TDD在图像分类和文本生成任务上均取得优异性能,大量消融研究验证了各设计组件的有效性。本工作建立了一种原理严谨的离散扩散方法,在保持扩散模型核心特性的同时,实现原生离散空间操作。
原文摘要 · Abstract (English)
While diffusion models excel at generating continuous data such as images, adapting them to discrete tasks has relied on indirect approaches that either operate in continuous embedding spaces or use token masking mechanisms, both of which deviate from modeling the true discrete data distribution that can be theoretically guaranteed by Tweedie's formula. We propose in-situ Tweedie Discrete Diffusion (TDD), a framework that performs diffusion guaranteed by Tweedie's formula directly within the discrete one-hot space, hence "in-situ." Unlike prior methods that diffuse continuous embeddings or mask tokens, TDD directly corrupts one-hot vectors with Gaussian noise and performs iterative denoising through a timestep-conditioned cross-entropy objective rather than mean-squared-error reconstruction. At each denoising step, the model predicts class probabilities, applies argmax to obtain discrete predictions, converts them to one-hot vectors, and feeds them into the next iteration with progressively reduced noise. This process naturally unifies discriminative classification and generative modeling under a single framework. Experiments demonstrate that TDD achieves strong performance on both image classification and text generation tasks, with extensive ablation studies confirming the effectiveness of each design component. Our work establishes a principled approach to discrete diffusion that preserves the core characteristics of diffusion models while operating natively in discrete space.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。