连续扩散语言模型通过上下文感知解码器，显著提升生成质量。
CoDAR: Continuous Diffusion Language Models are More Powerful Than You Think
- 采用两阶段框架，全程保持嵌入空间连续，仅在最后用自回归解码器进行上下文感知的取整。
- 在LM1B和OpenWebText上，生成质量超越潜在空间扩散模型，接近优秀离散模型。
- 仅通过调节解码器温度即可灵活控制流畅性与多样性，实用性强。
我们研究了为何连续扩散语言模型（DLMs）尽管具有理想的连续生成动态，却仍落后于离散扩散方法。在受控的词元恢复实验中，我们发现词元舍入（即从去噪嵌入到词元的最终投影）是主要瓶颈。基于此，我们提出CoDAR（连续扩散结合上下文自回归解码器），一个两阶段框架：整个扩散过程保持在嵌入空间连续，同时训练一个上下文条件化的离散化模块——一个跨注意力于去噪嵌入序列的自回归Transformer解码器，实现上下文感知的词元舍入。在LM1B和OpenWebText上的实验表明，CoDAR显著提升了生成质量，超越了潜在扩散模型，并达到与强离散DLMs相当的水平，同时提供简单的解码器温度控制以调节流畅性与多样性权衡。
原文摘要 · Abstract (English)
We study why continuous diffusion language models (DLMs) have lagged behind discrete diffusion approaches despite their appealing continuous generative dynamics. Under a controlled token--recovery study, we identify token rounding, the final projection from denoised embeddings to tokens, as a primary bottleneck. Building on these insights, we propose CoDAR (Continuous Diffusion with Contextual AutoRegressive Decoder), a two--stage framework that keeps diffusion entirely continuous in an embedding space while learning a strong, context--conditional discretizer: an autoregressive Transformer decoder that cross--attends to the denoised embedding sequence and performs contextualized rounding to tokens. Experiments on LM1B and OpenWebText demonstrate that CoDAR substantially improves generation quality over latent diffusion and becomes competitive with strong discrete DLMs, while exposing a simple decoder--temperature knob to navigate the fluency--diversity trade off.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。