arXiv:2607.26504cs.LG2026-07

让语言模型像程序员一样自由跳转推理,突破传统顺序生成限制。

From Interface to Inference: Eliciting Any-Order Inference from Any-Order Models

论文配图:From Interface to Inference: Eliciting Any-Order Inference from Any-Order Models
图 1 · 摘自论文原文
  • 用插入式掩码扩散模型打破固定位置束缚,支持跨不连续区域生成。
  • 将预测转移到粗粒度语义片段的隐空间,实现生成顺序的灵活搜索。
  • 在代码生成和数学推理任务中显著提升性能,适合需要灵活推理的应用。

许多离散推理任务(如代码生成)本质上是非因果的:程序员在高层结构与局部细节间来回切换,这种过程称为任意顺序推理。自回归语言模型缺乏原生的任意顺序接口,因此需手工设计机制才能实现非因果能力(如补全和下一次编辑预测)。能否设计出原生支持任意顺序推理的模型?掩码扩散模型因其任意顺序训练目标,成为有力候选。然而,其接口与推理能力之间存在差距,根源在于位置不确定性:固定画布的标记级模型可能知道应出现的语义成分,却不知如何定位。为此,我们提出两种互补方法:(1) 基于插入的掩码扩散模型(以FlexMDM为基础),通过插入操作放松固定位置约束,支持跨不连续区域生成;(2) 隐空间掩码扩散模型将预测迁移到更粗粒度的语义片段,实现隐空间生成顺序搜索。我们在7B规模的FlexMDM上训练用于Python编程,在125M规模的LatentMDM上训练用于GSM8K任务,实验表明两种方法均能诱导出不同的任意顺序推理行为,并提升下游任务表现。代码已开源:https://github.com/SeunggeunKimkr/genuine-any-order。

原文摘要 · Abstract (English)

Many discrete reasoning tasks, such as code generation, are inherently non-causal: programmers move between high-level structure and local details, a process we call any-order inference. For autoregressive language models, which lack a native any-order interface, non-causal abilities such as infilling and next-edit prediction require hand-designed mechanisms. Can we instead design models that natively support any-order inference? Masked diffusion models have recently emerged as compelling candidates, as their any-order training objective naturally offers an any-order prediction interface. This interface, however, does not automatically yield any-order inference. We demonstrate that this interface-inference gap stems from positional uncertainty: fixed-canvas, token-level models may know what semantic component should appear without knowing where to place it. In light of this, we propose two complementary approaches: (1) Insertion-based masked diffusion, building on FlexMDM (Kim et al, 2025), relaxes fixed-position commitments via insertions, enabling generation across non-contiguous regions. (2) Latent-space masked diffusion shifts prediction to coarser semantic segments, enabling search over latent generation orders. Empirically, we train a 7B FlexMDM for Python coding and a 125M LatentMDM for GSM8K and show that both approaches induce distinct any-order inference behaviors and improve downstream performance. We release our codebase at https://github.com/SeunggeunKimkr/genuine-any-order.

任意顺序推理掩码扩散代码生成隐空间建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。