通过引入隐变量提升离散生成中令牌依赖建模能力
Variational Masked Diffusion Models
- 在掩码扩散框架中加入隐变量以显式建模令牌间依赖
- 在数独和文本数据上显著提升生成一致性和依赖感知度
- 适合关注离散序列生成质量与全局结构的开发者
掩码扩散模型近期成为离散生成建模的灵活框架，但标准方法难以有效捕捉同时预测的令牌之间的依赖关系，导致依赖重要时生成质量下降。为此，我们提出变分掩码扩散（VMD），在掩码扩散过程中引入隐变量以显式建模令牌间依赖。在合成数据集上的控制实验表明，VMD成功学习到传统方法无法捕捉的依赖关系。我们在数独谜题和文本数据集上进一步验证了该方法的有效性，发现对令牌间依赖的学习显著提升了全局一致性。在多个领域中，VMD均提升了生成质量与依赖感知能力，凸显了将变分推断融入掩码扩散的价值。代码已开源：https://riccizz.github.io/VMD。
原文摘要 · Abstract (English)
Masked diffusion models have recently emerged as a flexible framework for discrete generative modeling. However, a key limitation of standard masked diffusion is its inability to effectively capture dependencies among tokens that are predicted concurrently, leading to degraded generation quality when dependencies among tokens are important. To explicitly model dependencies among tokens, we propose Variational Masked Diffusion (VMD), a framework that introduces latent variables into the masked diffusion process. Through controlled experiments on synthetic datasets, we demonstrate that VMD successfully learns dependencies that conventional masked diffusion fails to capture. We further validate the effectiveness of our approach on Sudoku puzzles and text datasets, where learning of dependencies among tokens improves global consistency. Across these domains, VMD enhances both generation quality and dependency awareness, highlighting the value of integrating variational inference into masked diffusion. Our code is available at: https://riccizz.github.io/VMD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。