提出连续语言生成新方法，用嵌入空间扩散提升文本质量与效率
ELF: Embedded Language Flows
- 在嵌入空间中进行连续扩散，最后才映射到离散词元
- 生成质量超越现有模型，采样步数更少
- 可直接复用图像领域技术，适合追求高效生成的研究者
扩散模型和基于流的模型已成为生成连续数据（如图像、视频）的主流方法。其成功促使研究者尝试将其应用于语言建模。然而，当前领先的扩散语言模型（DLMs）仍主要在离散词元空间操作。本文表明，通过最小调整即可使连续DLM在离散域中有效运行。我们提出嵌入语言流（ELF），一种基于连续时间流匹配的连续嵌入空间扩散模型。与现有DLM不同，ELF在多数时间步保持在连续嵌入空间，仅在最后一步通过共享权重网络映射为离散词元。该设计使得可直接适配图像领域成熟技术，如无分类器引导（CFG）。实验显示，ELF显著优于领先的离散与连续DLM，在更少采样步数下实现更高生成质量，表明其是实现高效连续语言建模的有前景路径。
原文摘要 · Abstract (English)
Diffusion and flow-based models have become the de facto approaches for generating continuous data, e.g., in domains such as images and videos. Their success has attracted growing interest in applying them to language modeling. Unlike their image-domain counterparts, today's leading diffusion language models (DLMs) primarily operate over discrete tokens. In this paper, we show that continuous DLMs can be made effective with minimal adaptation to the discrete domain. We propose Embedded Language Flows (ELF), a class of diffusion models in continuous embedding space based on continuous-time Flow Matching. Unlike existing DLMs, ELF predominantly stays within the continuous embedding space until the final time step, where it maps to discrete tokens using a shared-weight network. This formulation makes it straightforward to adapt established techniques from image-domain diffusion models, e.g., classifier-free guidance (CFG). Experiments show that ELF substantially outperforms leading discrete and continuous DLMs, achieving better generation quality with fewer sampling steps. These results suggest that ELF offers a promising path toward effective continuous DLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。