arXiv:2510.06190cs.LG2025-10被引 7

提出可自修正的任意流程生成,让语言模型更高效解决复杂推理问题。

On Powerful Ways to Generate: Autoregression, Diffusion, and Beyond

  • 引入任意流程生成,支持重掩码、插入和删除 token
  • 实验证明能处理传统模型无法解决的复杂推理任务
  • 适合需要动态修改与非顺序生成的场景,如编程与科学建模

扩散语言模型作为自回归模型的有力替代,具备并行和任意顺序生成能力,提升效率与灵活性。本文形式化研究掩码扩散模型(MDM)是否能突破自回归模型(ARM)的计算极限。结果表明,当上下文长度足够时,MDM 在解码步数上达到 PRAM 模型的最优并行时间复杂度,具有计算通用性。然而在控制变量后,仅靠任意顺序生成并未扩展 ARM 的求解能力。为此,提出新型生成范式——任意流程生成,通过支持重掩码、插入与删除,实现自修正、长度可变编辑与自适应并行。理论与实验均证明该方法可扩展至现有模型难以处理的复杂推理任务,并在代码生成、科学建模等非序列演化场景中至关重要。

原文摘要 · Abstract (English)

Diffusion language models have recently emerged as a competitive alternative to autoregressive language models. Beyond next-token generation, they are more efficient and flexible by enabling parallel and any-order token generation. However, despite empirical successes, their computational power and fundamental limitations remain poorly understood. In this paper, we formally study whether non-autoregressive generation in Masked Diffusion Models (MDM) enables solving problems beyond the reach of Auto-Regressive Models (ARM). Our results show that MDM with sufficiently large context length is computationally universal with decoding steps matching the optimal parallel time complexity in PRAM. However, when controlling for other factors, MDM's flexibility to generate in any-order does not expand what ARM can already solve. To address this, we propose a new form of generation called any-process generation, which extends MDM with capabilities to remask, insert and delete tokens, allowing self-correction, length-variable editing, and adaptive parallelism. Theoretically and empirically, we demonstrate these capabilities enable scalability to significantly harder reasoning problems that are otherwise intractable for ARM and vanilla MDM. Additionally, they prove essential for generation tasks where objects naturally evolve through non-sequential processes, crucial for extending current LLMs beyond natural language to domains such as coding and science.

扩散模型生成范式推理增强自修正

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。