连续扩散模型首次在语言建模上追平离散模型，实现高质量生成。
LangFlow: Continuous Diffusion Rivals Discrete in Language Modeling
- 通过Bregman散度将嵌入空间扩散与流匹配连接，构建新框架。
- 在LM1B和OpenWebText上分别达到30.0和24.6的困惑度，媲美顶尖离散模型。
- 创新噪声调度与自条件训练，适合追求高效可控生成的研究者。
连续扩散模型在图像等多模态生成中表现优异，但在语言建模领域因数据稀疏与设计空间未充分探索，长期落后于离散模型。本文提出LangFlow，首个可媲美离散扩散的语言建模框架，通过Bregman散度将嵌入空间扩散与流匹配结合，并引入三项关键创新：(1) 推导基于ODE的负对数似然（NLL）界，实现连续流模型的严谨评估；(2) 提出信息均匀原则，设计基于Gumbel分布的可学习噪声调度器；(3) 重构训练协议，引入自条件机制，显著提升嵌入空间模型的似然与样本质量，效果不同于离散扩散。实验表明，LangFlow在LM1B上达到30.0的困惑度（PPL），在OpenWebText上达24.6，同时在7个基准中的4个零样本迁移任务超越自回归基线，首次证明连续扩散在语言建模中的可行性与优势。
原文摘要 · Abstract (English)
Continuous diffusion has been the foundation of high-fidelity, controllable, and few-step generation of many data modalities such as images. However, in language modeling, prior continuous diffusion language models (DLMs) lag behind discrete counterparts due to the sparse data space and the underexplored design space. In this work, we close this gap with LangFlow, the first continuous DLM to rival discrete diffusion, by connecting embedding-space DLMs to Flow Matching via Bregman divergence, alongside three key innovations: (1) we derive a novel ODE-based NLL bound for principled evaluation of continuous flow-based language models; (2) we propose an information-uniform principle for setting the noise schedule, which motivates a learnable noise scheduler based on a Gumbel distribution; and (3) we revise prior training protocols by incorporating self-conditioning, as we find it improves both likelihood and sample quality of embedding-space DLMs with effects substantially different from discrete diffusion. Putting everything together, LangFlow rivals top discrete DLMs on both the perplexity (PPL) and the generative perplexity (Gen. PPL), reaching a PPL of 30.0 on LM1B and 24.6 on OpenWebText. It even exceeds autoregressive baselines in zero-shot transfer on 4 out of 7 benchmarks. LangFlow provides the first clear evidence that continuous diffusion is a promising paradigm for language modeling. Homepage: https://github.com/nealchen2003/LangFlow
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。