通过引入隐变量空间,提升离散扩散模型的生成效率与质量。
Latent-Augmented Discrete Diffusion Models
- 用可学习的隐变量扩展令牌空间,建模跨标记依赖关系。
- 在低采样预算下优于当前最优基线,尤其适合每步去噪多个标记。
- 支持连续与离散隐变量,灵活适配不同生成场景。
离散扩散模型已成为快速语言生成的有力工具,但现有实现多依赖因子化反向过程,忽略跨标记依赖,影响少步生成性能。本文提出潜变量增强的离散扩散模型(LADD),引入可学习的辅助潜变量通道,在(令牌, 潜变量)联合空间中进行扩散。潜变量提供表达联合结构的中间表示,同时保持参数化的可处理性。我们构建了连续潜变量(Co-LADD)和离散潜变量(Di-LADD)版本,并研究两种推理调度:联合扩散(同时去噪数据与潜变量)和顺序扩散(先解潜变量,再条件采样令牌)。推导了类似ELBO的目标函数,分析了潜变量表达力与扩散兼容性之间的权衡。实验表明,相较于当前最优的掩码离散扩散基线,LADD在无条件生成指标上均有提升,且在低采样预算下表现更优,特别适用于每步去噪多个标记的场景。
原文摘要 · Abstract (English)
Discrete diffusion models have emerged as a powerful class of models and a promising route to fast language generation, but practical implementations typically rely on factored reverse transitions ignoring cross-token dependencies and degrading few-step performance. We propose Latent-Augmented Discrete Diffusion (LADD), which introduces a learnable auxiliary latent channel and performs diffusion over the joint (token, latent) space. The latent variables provide an intermediate representation expressing joint structure while preserving tractable parameterizations. We instantiate LADD with continuous latents (Co-LADD) and discrete latents (Di-LADD), and study two inference schedules: a joint diffusion that denoises data and latents together, and a sequential diffusion that first resolves latents and then samples tokens conditionally. We derive ELBO-style objectives and analyze design choices that balance latent expressivity with diffusion compatibility. In experiments, LADD models yield improvements on unconditional generation metrics as compared to state-of-the-art masked discrete diffusion baselines, and are effective at lower sampling budgets, where unmasking many tokens per step is desirable.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。