arXiv:2601.04854cs.CLcs.AI2026-01被引 1

用连续向量预测代替逐词选择,生成更灵活的文本。

Projected Autoregression: Autoregressive Language Generation in Continuous State Space

  • 用嵌入空间的连续预测替代离散选词,仅在提交时才转为词语。
  • 即使立即提交,生成效果也优于传统方法,且支持迭代优化。
  • 适合研究生成机制、可控生成或想突破传统模型框架的人。

标准自回归语言模型通过逐次选择离散下一个词来生成文本,每一步都伴随不可逆的决策。本文提出一种新范式:投影自回归(Projected Autoregression)。该方法将词的选择替换为在嵌入空间中的连续预测,仅在提交时刻进行最近邻投影得到离散词。模型通过回归与对比学习目标预测下一词向量,而离散词仅作为下游接口出现。可选的可变后缀(“液态尾部”)支持提交前的迭代优化,但核心创新在于:下一步预测是连续的,离散词仅在最后生成。该方法建立了一种替代离散选词的自回归接口:语言生成可围绕连续状态预测展开,并延迟离散承诺。优化局限于左到右因果过程中的短因果后缀,而非全序列去噪。这一设计带来两方面结果:其一,形成独特生成模式——即使立即投影(K=1),连续预测生成的文本结构与动态也显著区别于测试过的基准模型,包括计算量匹配的最佳16个重排序基线;其二,揭示了自回归生成中的连续控制面:方向速率、历史噪声、延迟承诺、状态空间引导及嵌入几何可直接作用于生成状态,直至词提交前。这些发现将重复词选择纳入更广的自回归接口家族,并揭示连续状态空间是语言生成的更大算法设计空间。

原文摘要 · Abstract (English)

Standard autoregressive language models generate text by repeatedly selecting a discrete next token, coupling prediction with irreversible commitment at every step. We show that token selection is not the only viable autoregressive interface. \textbf{Projected Autoregression} replaces token selection with continuous prediction in embedding space followed by discrete projection at commitment time. The model predicts next-token vectors via regression and contrastive objectives, while discrete tokens arise only by nearest-neighbor projection. An optional mutable suffix (``liquid tail'') enables iterative refinement before commitment, but the central change is more basic: next-step prediction is continuous, and discrete tokens are produced only as a downstream interface. Projected Autoregression establishes a concrete alternative to token-selection autoregression: language generation can be organized around continuous-state prediction with delayed discrete commitment. Refinement remains local to a short causal suffix within a left-to-right causal process, rather than a sequence-wide denoising process. This separation has two consequences. First, it induces a \emph{distinct generation regime}: even with immediate projection ($K{=}1$), continuous prediction yields text structure and dynamics that differ from tested token-space AR baselines, including a compute-matched best-of-16 reranking baseline. Second, it exposes a \emph{continuous control surface} inside autoregressive generation: direction rate, history noise, delayed commitment, state-space guidance, and embedding geometry act directly on the evolving generative state before token commitment. Taken together, these results place repeated token selection within a larger family of autoregressive interfaces and expose continuous state space as a broader algorithmic design space for language generation.

自回归连续生成语言模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。