arXiv:2608.02602cs.CL2026-08

用连续潜空间建模文本,实现高保真语言生成。

AURORA-LM: Autoencoding Unified Representation for Continuous-Latent Diffusion Language Modeling

论文配图:AURORA-LM: Autoencoding Unified Representation for Continuous-Latent Diffusion Language Modeling
图 1 · 摘自论文原文
  • 构建可解码的高容量连续文本表示,分离表示与分布建模。
  • 在OpenWebText和XSum任务上超越现有连续语言模型表现。
  • 适合追求高精度生成与潜在空间建模的研究者使用。

语言生成在生成建模中仍属例外:图像、视频和音频越来越多地采用连续潜空间建模,而文本生成仍主要依赖离散词元。现有连续语言模型或沿用非联合生成设计的嵌入空间,或压缩自编码潜变量以简化扩散过程,牺牲词级保真度。本文不简化表示以适应生成模型,而是保留高容量、可解码的文本潜变量,并设计扩散模型直接学习其分布。提出AURORA-LM,一种连续潜变量扩散语言模型,将文本表示构建与分布建模解耦。基于查询的编码器-解码器生成前缀对齐的高容量潜序列,块因果扩散变换器通过流匹配学习分布,左到右生成块,同时并行去噪块内位置。由于该潜变量更难建模,AURORA-LM仅限制噪声输入路径,保留完整清洁潜变量预测目标,无需降低解码器容量。进一步校准噪声水平分布至潜变量宽度,并引入自轨迹一致性,弥合训练噪声独立采样与推理迭代去噪的差异。在Ascend NPUs上实验,于OpenWebText自由生成与XSum摘要任务上表现最佳。10亿参数规模(约1500 EFLOPs计算量)下进一步提升性能,超越更大公开的潜扩散语言模型,在匹配评估协议下取得领先。所有实验均在Ascend NPU上完成。

原文摘要 · Abstract (English)

Language remains an outlier in generative modeling: while images, video, and audio are increasingly modeled in continuous latent spaces, text generation still relies predominantly on discrete tokens. Existing continuous language models either inherit embedding spaces not designed for joint generation and decoding, or compress autoencoded latents to ease diffusion, sacrificing token-level fidelity. Instead of simplifying the representation to suit the generative model, we preserve a high-capacity, decodable text latent and design the diffusion model to learn its distribution directly. We introduce AURORA-LM, a continuous-latent diffusion language model that separates the construction of a decodable text representation from the modeling of its distribution. A Query-based Encoder-Decoder organizes text into a high-capacity, prefix-aligned latent sequence, and a Block-causal Diffusion Transformer learns its distribution through flow matching, generating blocks left to right while denoising positions within each block in parallel. Because such a latent is harder for diffusion to model, AURORA-LM restricts only the noisy-input pathway while retaining the full clean-latent prediction target, accommodating full-width latents without reducing decoder-facing capacity. We further calibrate the noise-level distribution to the latent width, and introduce self-trajectory consistency to bridge independently sampled training noise and iterative denoising at inference. AURORA-LM achieves the strongest performance among evaluated continuous and diffusion-based language models on OpenWebText free generation and XSum summarization. Scaling to 1B parameters with about 1500 EFLOPs of total compute yields further gains, surpassing a larger publicly released latent-diffusion language model under a matched evaluation protocol. All experiments are conducted on Ascend NPUs.

连续表示扩散模型语言生成潜空间建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。