统一了离散扩散模型的构建框架，揭示其核心设计权衡。
Discrete Diffusion Models: A Unified Framework from Tokenization to Generation
- 从分词、词汇拓扑到领域字母表，系统构建离散状态空间
- 揭示不同方法在训练、推理、扩展上的共性权衡
- 为未来研究提供可拓展的设计方向，适合模型架构研究者
离散去噪扩散模型（DDMs）近期成为离散数据建模的有力替代方案，相比自回归模型具备并行生成与迭代全局优化能力。与连续扩散不同，DDMs 的本质取决于离散状态空间的构建方式：分词方案、词汇拓扑结构以及领域特定的结构字母表。本文提出一个统一的概念框架，将离散扩散模型视为底层离散状态空间的构建过程。在此框架下，现有的多种形式——包括转移矩阵、掩码/吸收态、以及基于得分/比值的方法——均被视为同一设计空间的不同实例。该框架进一步揭示了训练目标、推理算法、扩展行为、系统优化和评估协议之间的共性设计权衡，指出了若干有前景的未来研究方向。
原文摘要 · Abstract (English)
Discrete denoising diffusion models (DDMs) have recently emerged as a compelling alternative to autoregressive (AR) modeling for discrete data, offering parallel generation and iterative global refinement capabilities. Unlike continuous diffusion, where the state space is fixed, DDMs are fundamentally shaped by how the discrete state space is constructed: the tokenization scheme, the vocabulary topology, and domain-specific structural alphabets. This work introduces a unified conceptual framework that views discrete diffusion models through the construction of the underlying discrete state space. Within this framework, existing formulations, including transition-matrix, masking/absorbing-state, and score/ratio-based approaches, emerge as different instantiations of a common design space. The framework further exposes common design trade-offs across training objectives, inference algorithms, scaling behavior, systems optimization, and evaluation protocols, suggesting several promising directions for future research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。