arXiv:2605.11125cs.LG2026-05被引 6

用球面流提升语言建模,实现高效并行生成。

Language Modeling with Hyperspherical Flows

  • 在单位球面上通过旋转向量生成文本,避免显式编码词表。
  • 在数学与代码等推理任务上显著优于现有连续流模型。
  • 适合需要高质量并行生成的高阶语言任务研究者。

离散扩散语言模型作为自回归模型的替代方案,因其并行生成能力发展迅速。然而,为保证可计算性,这类模型通常采样自分解分布,表达能力较弱。最近的流语言模型(FLMs)将连续流应用于语言建模,通过确定性微分方程从噪声映射到数据,避免了分解采样。但传统FLMs基于独热向量,其维度随词汇量线性增长,训练成本高昂。且所有独热向量在ℓ₂空间中等距,加高斯噪声缺乏明确语义意义(不同于图像中逐步破坏结构)。本文提出$ℝ$-FLM,一种定义在超球面$ℝ^{d-1}$上的隐变量流语言模型。该模型通过学习速度场在球面上旋转向量生成序列,无需显式构建独热向量。此前的FLMs在生成困惑度(Gen. PPL)上达到与自回归模型相当水平,但高概率样本未必在数学和代码等可验证领域正确。$ℝ$-FLM在大词汇量推理任务上显著提升连续流语言模型性能,在标准温度采样(T=1)下接近掩码扩散模型表现,但在优化低温度(T=0.1)解码时仍存在差距。

原文摘要 · Abstract (English)

Discrete Diffusion Language Models progressed rapidly as an alternative to autoregressive (AR) models, motivated by their parallel generation abilities. However, for tractability, discrete diffusion models sample from a factorized distribution, which is less expressive than AR. Recent Flow Language Models (FLMs) apply continuous flows to language, transporting noise to data with a deterministic ODE that avoids factorized sampling. FLMs operate on one-hot vectors whose dimension scales with the vocabulary size, making FLMs costly to train. Moreover, since all distinct one-hot embeddings are equidistant in $\ell_2$, adding Gaussian noise does not have a clear semantic interpretation (unlike images, where Gaussian noise progressively degrades structure). We introduce $\mathbb{S}$-FLM, a latent FLM in the hypersphere. $\mathbb{S}$-FLM generates sequences by rotating vectors in $\mathbb{S}^{d-1}$ along a velocity field learned with cross-entropy, avoiding the overhead of materializing one-hot vectors. Previous FLMs match AR in Generative Perplexity (Gen.\ PPL), but samples with high likelihood are not necessarily correct in verifiable domains such as math and code. $\mathbb{S}$-FLM substantially improves continuous flow language models on large-vocabulary reasoning and closes the gap to masked diffusion under standard-temperature sampling ($T=1$), while a gap remains under optimized low-temperature ($T=0.1$) decoding.

语言建模流模型球面流并行生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。