arXiv:2605.07933cs.CL2026-05被引 3

让文生文的扩散模型同时学好潜空间,生成更快更准。

How to Train Your Latent Diffusion Language Model Jointly With the Latent Space

  • 三模块联合训练,自动构建易还原的连续文本潜空间。
  • 在OpenWebText和LM1B上比现有模型快2-13倍,效果更优。
  • 提出温启动、自适应采样等技巧,显著提升生成质量。

潜空间扩散模型通过操作连续文本表示并并行去噪整句,为非自回归文本生成提供了替代离散扩散的方案。其核心挑战在于构建合适的潜空间。本文提出潜空间扩散语言模型(LDLM),将潜编码器、扩散模型与解码器联合训练。LDLM通过可学习编码器重塑预训练语言模型的表征,生成易于去噪且可解码为词元的潜变量。我们发现直接联合训练会导致扩散模型质量低下,因此提出包含MSE解码损失、扩散到编码器温启动、自适应时间步采样及解码器输入噪声的简单训练方案。消融实验表明各组件均显著影响生成性能。在OpenWebText和LM1B数据集上,LDLM生成效果优于现有离散与连续扩散语言模型,且速度提升2–13倍,证明联合学习潜空间是使潜空间扩散模型在文本生成中具备竞争力的关键一步。

原文摘要 · Abstract (English)

Latent diffusion models offer an attractive alternative to discrete diffusion for non-autoregressive text generation by operating on continuous text representations and denoising entire sequences in parallel. The major challenge in latent diffusion modeling is constructing a suitable latent space. In this work, we present the Latent Diffusion Language Model (LDLM), in which the latent encoder, diffusion model, and decoder are trained jointly. LDLM builds its latent space by reshaping the representations of a pre-trained language model with a trainable encoder, yielding latents that are easy to both denoise and decode into tokens. We show that naive joint training produces a low-quality diffusion model, and propose a simple training recipe consisting of an MSE decoder loss, diffusion-to-encoder warmup, adaptive timestep sampling, and decoder-input noise. Ablations show that each component substantially impacts generation performance. On OpenWebText and LM1B, LDLM achieves better generation performance than existing discrete and continuous diffusion language models while being $2{\text -}13\times$ faster, indicating that jointly learning the latent space is a key step toward making latent diffusion competitive for text generation.

扩散模型文本生成潜空间联合训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。