将语言生成从离散转向连续语义空间，提升质量与速度。
CRoCoDiL: Continuous and Robust Conditioned Diffusion for Language
- 用连续语义空间替代离散分布，改进掩码扩散模型
- 在无条件生成中实现超10倍加速且质量更优
- 适合追求高效高质文本生成的研究者
掩码扩散模型（MDMs）为非自回归生成提供了高效方案，但常因依赖离散边际分布而面临词元依赖建模不足与语义不连贯问题。本文提出CRoCoDiL（连续且鲁棒的语言条件扩散），通过将扩散过程迁移至连续句级语义空间，联合训练编码器-去掩码器架构，使去掩码操作基于连续潜在表示进行。由此构建新型自编码器，解码由MDM算法完成。在此框架下，进一步提出两种无条件文本生成算法：连续-然后-离散（ConThenDisc），先在连续空间生成潜在表示再转为词元；连续-在-离散内（ConWithinDisc），在离散采样过程中持续优化潜在表示。基于LLaDA的实验表明，该方法在无条件生成中实现超过10倍的采样速度提升，并取得更优生成质量。
原文摘要 · Abstract (English)
Masked Diffusion Models (MDMs) provide an efficient non-causal alternative to autoregressive generation but often struggle with token dependencies and semantic incoherence due to their reliance on discrete marginal distributions. We address these limitations by shifting the diffusion process into a continuous sentence-level semantic space. We propose CRoCoDiL (Continuous and Robust Conditioned Diffusion for Language), a unified fine-tuning approach that jointly trains an encoder-demasker architecture, grounding the MDM demasking in continuous latent representations. This leads to the formation of a novel autoencoder in which decoding is obtained by an MDM algorithm. Relying on the same framework, we introduce two unconditional text synthesis algorithms: Continuous-Then-Discrete (ConThenDisc), a hybrid-diffusion approach that first generates latent representations in continuous space and then decodes these to tokens via an MDM, and Continuous-Within-Discrete (ConWithinDisc), a multi-diffusion strategy that refines latent representations throughout the discrete sampling process. Experiments using LLaDA show that our methods achieve superior generation quality and more than 10x faster sampling speeds in an unconditional setting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。