通过连续潜在空间提升语言扩散模型采样质量与速度
DiLaDiff: Distilled Latent-Augmented Diffusion for Language Modeling
- 用自编码器构建语义连续潜在空间，改进扩散语言模型
- 无需蒸馏即加速推理，蒸馏后生成潜变量几乎无耗时
- 适合需要高效高质量文本生成的场景
扩散语言模型固有缺陷是难以捕捉解码词元间的相关性，导致采样质量与吞吐量之间存在严峻权衡。为此，我们提出DiLaDiff，一种带有三个组件的掩码扩散语言模型变体：（1）由预训练掩码扩散语言模型微调得到的具备语义能力的连续潜在空间；（2）学习编码器分布先验的潜在扩散模型；（3）将学习到的先验蒸馏为少步潜在生成模型的一致性模型。实验表明，即使不使用蒸馏，我们的潜在引导扩散模型也优于掩码扩散基线，并显著加速推理。一致性蒸馏进一步降低连续扩散的计算开销，使得潜变量生成时间可忽略不计，相比离散解码大幅提速。
原文摘要 · Abstract (English)
Diffusion language models intrinsically fail to capture correlations between decoded tokens, which leads to a harsh trade-off between sampling quality and throughput. To solve this issue, we propose DiLaDiff, a variant of masked diffusion language models with three components: (1) a continuous latent space with semantic capabilities, learned by an auto-encoder fine-tuned from an existing masked diffusion language model; (2) a latent diffusion model learning the prior over the encoder distribution; (3) a consistency model distilling the learned prior into a few-step latent generative model. We show that, even without distillation, our latent-guided diffusion model outperforms the masked diffusion baseline while significantly accelerating inference. Consistency distillation further lowers the computational overhead of continuous diffusion, such that the latent is generated in negligible time compared to discrete decoding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。